AMD calls SYSmark 2012 benchmark biased, quits BAPCo consortium

James Delahunty
23 Jun 2011 17:09

Nvidia and VIA follow AMD out of BAPCo consortium.
The Business Applications Performance Corporation (BAPCo) is a non-profit consortium tasked with developing performance benchmarks for computer systems based on widely used software applications and operating systems. BAPCo has been developing the SYSmark 2012 benchmark intended to be used by businesses and governments to make spending decisions.
On Tuesday (June 21), AMD publicly announced that it has left the BAPCo consortium, and said it cannot endorse the SYSmark 2012 benchmark. AMD's Senior Vice President and Chief Marketing Officer, Nigel Dessau, had questioned some aspects of the benchmark in a blog post last year titled "Have you Paid the 'SYSmark Tax'?", in which he estimated that government and educational institutions had needlessly spent over $7 billion during the preceding five years as a result of the SYSmark benchmark.
According to Dessau, BAPCo's response to the blog was a threat to expel AMD from the consortium. Nowadays most of us use applications that run HTML 5, use Javascript and other elements that benefit greatly from GPU acceleration. Internet Explorer 9, Microsoft Office, Windows Movie Maker and Adobe Flash are examples of applications given by AMD that benefit from GPU acceleration and are widely used.
The benchmarks then should be relevant to the applications that are used by people depending on the benchmark scores. AMD's major complaint is that SYSMark is comprised of unrepresentative workloads, ignoring the importance of heterogeneous computing in today's environment. It also accuses SYSmark of generating misleading results that lead to poor purchasing decisions.
While AMD clearly had issues with the SYSmark 2012 development, it stuck around to attempt to change it to a more open, transparent and processor-neutral benchmark, and did get some benchmarks introduced that it said properly represented what people do in a day, "instead of 35,000 line spreadsheets."
However, there still comes the issue of the weighting given to the real-world benchmarks, as the weighting is what really determines the score. Here is where AMD decided SYSmark 2012 was not properly representing the modern enviroment, saying it continues to distort workload performance and offers less transparency to users.
Nigel Dessau summed up the problems with the SYSmark 2012 benchmark with three main points...

• While SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score.
• In fact, a relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.
• And SM2012 doesn't represent the evolution of computer processing and how that evolution is influencing average users' experience. SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today's business environments.

AMD's decision to quit BAPCo was followed since by Nvidia and VIA. AMD is now considering other options, including the creation of a new consortium to establish a truly open benchmark that would measure overall system performance.
While the core of AMD's complaint does imply bias toward Intel, Dessau says that this is not the core reason for AMD's breaking from the BAPCo consortium, again, giving three reasons...

• It's about fairness. Fairness to consumers and business users, to governments and other organizations that make purchasing decisions based on benchmarks, and, in the case of SYSmark, needlessly overspend because of it.
• It's about relevance. Because do you want to buy a system based on an outdated approach to measuring performance? Don't you want your system's performance measured against relevant measures like HTML5 or GPU acceleration? And shouldn't a benchmark that measures PC performance be relevant to other devices that are likely in your life (if you're reading this blog I think it's safe to presume you use an array of devices – I do). Benchmarks should measure the way people engage with their devices today – not stick to a formula more appropriate for the last millennium.
• And it's about openness. Because you, and IT purchasing managers, should know what a benchmark represents and what the score really means to how the device will be used. That's being set free.

BAPCo responded to AMD's public flogging by saying that all BAPCo members were given one vote on any proposals made by member companies, and that AMD voted in favor of SYSmark 2012 development milestones 80 percent of the time. It also said that AMD was supported in all of the SYSmark 2012 proposals put forth.