Date:  5 January 1983 17:42 mst
From:  Lippard.DSOP (James J. Lippard)
Subject:  Miscellaneous Digest V2 #3
Reply-To:  misc
To:   (Miscellaneous Mailing List)

Miscellaneous Digest                              Volume 2 : Issue 3

Today's topics:
		Benchmarks ...
---------------------------------------------------------------------
Date:  5 January 1983 10:02 mst
From:  Jerry Crow
Subject:  Benchmarks ...

Benchmarks,  viewed  from  either  side (potential customer or vendor), are
interesting  events.   I was a member of the benchmark team which conducted
vendor evaluations for a procurement specification issued by the University
of  Kansas.  We visited IBM, Honeywell, Sperry (still UNIVAC at that time),
and Control Data.

The  thing that made the greatest impression on me was the time the vendors
had  obviously  spent  in preparing for the benchmark.  Our benchmark was a
~serious~  benchmark,  and  was  acknowledged  by all vendors to be so (IBM
personnel told us it was the toughest non-Federal benchmark they had ~ever~
done),  so  some  preparation  was to be expected.  I hadn't really thought
much  about  such  things prior to our trips though, and I guess I had this
vague  notion  that  "benchmark" was a euphemism for "bring in the customer
who  has  already made up his mind to go with your equipment, let him see a
few  printers  generate  some  output,  and  then move the remainder of the
benchmark  activities to a more convivial location, at which you ensure the
prospect  is  supplied  with  large  quantities  of  food  and  even vaster
quantities  of  beverages,  at ~your~ expense, of course".  Well, it wasn't
that way, folks.  Well ...  we ate some free food and drank a few pops, and
at vendor expense (it would have been rude to refuse), true, but, as I said
above,  it  was  obvious  that  a  considerable  amount  of preparation had
preceded our arrival.

The  benchmark  activities  themselves consumed, on the average, 15 hours a
day  at  IBM and about 12.5 hours a day at Honeywell, to cite two examples.
It  was  a tough job friends.  And I suspect the vendors worked even harder
than we did.

Enough  ...   Craig talking about all the work Ed did triggered my memories
of  the benchmark era: the last ten weeks of 1975 during which time, except
for  the  Thanksgiving  and Christmas holidays, I spent two nights at home.
It  was  a  very  valuable experience.  I gained a much better idea of what
serious benchmarking is all about.

One thing though: benchmarks are like EPA mileage ratings.  In the absolute
sense,  the  information they convey bears little semblance to reality.  In
the  relative  sense,  they  are excellent barometers of comparative system
performance.
--------------------

End of Miscellaneous Digest
***************************