Are Apple's benchmarks misleading?


Another Stevenote, another controversy about benchmarks. An commentary by Tom Yager on InfoWorld states that "Apple used multiprocessor benchmarks to skew the performance advantage." I have some issues with this.

First, Apple has this time almost completely disclosed the information about their benchmarks, including compilers (IBM optimizing compilers vs. Intel Optimizing compilers, not the GCC compilers that have been rumored elsewhere) and specifics on which benchmarks were run (SPECint_rate_base2000 and SPECfp_rate_base2000).

Using the base version of the benchmarks (instead of the peak versions) indicates that a very limited number of compiler options were allowed for changing performance.

The assertion by Mr. Yager that the rate benchmarks are inherently multi- processor and therefore are not appropriate for the G5's that were compared in the iMac comparison is a bit off from my understanding of the benchmarks.

According to the SPEC site, the difference between the rate and speed benchmarks is as follows:

  • The SPEC speed metrics (e.g., SPECint2000) are used for comparing the ability of a computer to complete single tasks.
  • The SPEC rate metrics (e.g., SPECint_rate2000) measure the throughput or rate of a machine carrying out a number of tasks.

Now, if we were running DOS, it might be important to take a look at the speed benchmarks, but to look at user's workflow, there are three important things to keep in mind:

  1. Macintosh users often do more than one thing at a time
  2. OS X is a very efficient multi-tasking environment and utilizes preemptive multitasking
  3. Many Macintosh applications are already designed to take advantage of dual processors, as we have been using them on the Macintosh for years and expect to take advantage of them in applications such as Photoshop and Final Cut Pro.

In the end, Mr. Yager might be right, but I don't think it's as cut and dry as he believes and am looking forward to seeing the effect of these new dual-core machines in person.

My guess is, as I have seen with my dual-processor and quad-core machines in the past, I will be seeing an enjoyable increase in speed.