AfterDawn: Tech news

AMD calls SYSmark 2012 benchmark biased, quits BAPCo consortium

Written by James Delahunty (Google+) @ 23 Jun 2011 17:09 User comments (3)

AMD calls SYSmark 2012 benchmark biased, quits BAPCo consortium Nvidia and VIA follow AMD out of BAPCo consortium.
The Business Applications Performance Corporation (BAPCo) is a non-profit consortium tasked with developing performance benchmarks for computer systems based on widely used software applications and operating systems. BAPCo has been developing the SYSmark 2012 benchmark intended to be used by businesses and governments to make spending decisions.

On Tuesday (June 21), AMD publicly announced that it has left the BAPCo consortium, and said it cannot endorse the SYSmark 2012 benchmark. AMD's Senior Vice President and Chief Marketing Officer, Nigel Dessau, had questioned some aspects of the benchmark in a blog post last year titled "Have you Paid the 'SYSmark Tax'?", in which he estimated that government and educational institutions had needlessly spent over $7 billion during the preceding five years as a result of the SYSmark benchmark.

According to Dessau, BAPCo's response to the blog was a threat to expel AMD from the consortium. Nowadays most of us use applications that run HTML 5, use Javascript and other elements that benefit greatly from GPU acceleration. Internet Explorer 9, Microsoft Office, Windows Movie Maker and Adobe Flash are examples of applications given by AMD that benefit from GPU acceleration and are widely used.

The benchmarks then should be relevant to the applications that are used by people depending on the benchmark scores. AMD's major complaint is that SYSMark is comprised of unrepresentative workloads, ignoring the importance of heterogeneous computing in today's environment. It also accuses SYSmark of generating misleading results that lead to poor purchasing decisions.

While AMD clearly had issues with the SYSmark 2012 development, it stuck around to attempt to change it to a more open, transparent and processor-neutral benchmark, and did get some benchmarks introduced that it said properly represented what people do in a day, "instead of 35,000 line spreadsheets."

However, there still comes the issue of the weighting given to the real-world benchmarks, as the weighting is what really determines the score. Here is where AMD decided SYSmark 2012 was not properly representing the modern enviroment, saying it continues to distort workload performance and offers less transparency to users.

Nigel Dessau summed up the problems with the SYSmark 2012 benchmark with three main points...
  • While SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score.
  • In fact, a relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.
  • And SM2012 doesn't represent the evolution of computer processing and how that evolution is influencing average users' experience. SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today's business environments.
AMD's decision to quit BAPCo was followed since by Nvidia and VIA. AMD is now considering other options, including the creation of a new consortium to establish a truly open benchmark that would measure overall system performance.

While the core of AMD's complaint does imply bias toward Intel, Dessau says that this is not the core reason for AMD's breaking from the BAPCo consortium, again, giving three reasons...
  • It's about fairness. Fairness to consumers and business users, to governments and other organizations that make purchasing decisions based on benchmarks, and, in the case of SYSmark, needlessly overspend because of it.
  • It's about relevance. Because do you want to buy a system based on an outdated approach to measuring performance? Don't you want your system's performance measured against relevant measures like HTML5 or GPU acceleration? And shouldn't a benchmark that measures PC performance be relevant to other devices that are likely in your life (if you're reading this blog I think it's safe to presume you use an array of devices I do). Benchmarks should measure the way people engage with their devices today not stick to a formula more appropriate for the last millennium.
  • And it's about openness. Because you, and IT purchasing managers, should know what a benchmark represents and what the score really means to how the device will be used. That's being set free.
BAPCo responded to AMD's public flogging by saying that all BAPCo members were given one vote on any proposals made by member companies, and that AMD voted in favor of SYSmark 2012 development milestones 80 percent of the time. It also said that AMD was supported in all of the SYSmark 2012 proposals put forth.

Tags: AMD GPGPU
Previous Next  

3 user comments

124.6.2011 2:12

Most of the time benchmarks don't mean much, really you buy a GFX cards and they are pretty much all the same speeds, it's really just the features and what you really after that matters..

Same deal with CPUs Quad 6 and 8 cores are the norm now, 9 times out of 10 you can't push these CPUs to the max load anymore as they process so much stuff so quickly that it's very hard to max them out.

You could buy the most hardcore games today and it'll barely push these CPUs over 50%.

224.6.2011 11:23

@xtago... I agree... And I also agree with the vendors walking out on the consortium as well. It's been a long time coming. Big bloat money has been swaying statistics for years manipulating stocks & purchase power for eons. Sheeple will also follow suit because they won't/refuse to see in 'test driven' machines that performance may not actually be any different in the consumer market.

Everyone buys these testing software packages looking to test their machinery to see if their getting the best, when the software manufacturer wrote the code specifically to work on one CPU set only. It's biased from the get-go & was consigned by the best money could buy. Call it a reverse insider trading, which I'm sure is still illegal & any blind man could see this is what has been going on; further displayed by the aforementioned walk out, not my little tirade here.

In hopes of keeping this short, bullying corporations need to stop. Policing actions won't do it. Elected government sanctions won't do it. So I guess we the purchasing public will just have to pull from the pages of a few crappy science fiction novels & literally have the the corporate giants go to war with one another.

That or we really need for organized crime to get back in the saddle again so the government will start doing their jobs like their supposed to. But that's another story.


324.6.2011 18:54

I agree with this 100% Real world performance is what matters. Some may make fun of my Phenom X4 compared to their Core i7 but It performs very well in real world applications. Using Synthetic Benchmarks is mostly a marketing scheme and there has been so many times in the past that evidence is disclosed of these benchmarks being taylored or tweaked to point the advantage a certain way.


"Have you tried turning it off and on again?" ~ Roy Trenneman

http://www.facebook.com/BlueLightningTechnicalServices

Comments have been disabled for this article.

News archive