Samsung allegedly boosting benchmark performance

Anandtech digs deep on a forum allegation and finds depressing results.

Read the whole story

 

[Adam Starkey]

Firehawke[/url]":3rqyvvhj]ATi and nVidia were both caught in the past doing this with video card benchmarks, too. I particularly remember one incident involving Quake 3 benchmark optimizations about twelve years ago where ATi was doing some specific optimizations that would only trigger when it detected the calling application was "quake3.exe"

Made it really easy to prove, too. Just rename the EXE and you'd see performance drop considerably.

Yeah that was pretty douchey. but in a way not quite as obnoxious as this. At least in that case one could make the argument that a well written application *could* get that performance out of those video cards. With a combination of good drivers, APIs, and game engine code, those figures were potentially reachable. By contrast there's no way a Samsung device is going to run like the benchmark rigging suggests, as the devices would most likely run uncomfortably hot and suffer much higher rates of inside warranty failure.

All in all, asdf25's comment pretty much sums it up.

 

[Adam Starkey]

Firehawke[/url]":k088ww83]
I look at it this way: It definitely proves that benchmark cheating is widespread and that you should ALWAYS take this stuff with a grain-- perhaps even an entire shaker-- of salt. I do believe this is good reason to call Samsung on the carpet to explain themselves, though. There's no excuse for this kind of blatant bullshit.

I totally agree. The auto industry sending out ringers to magazines to test is pretty much the accepted norm. It's wrong, and if no-one calls them out on it, then there's no incentive for anyone to play fair.

Unfortunately, this'll rattle around the tech-sphere irritating a handful of nerds who already undertstand that those benchmarks are useful only as a cheap way of irritating other fanboys. The average Joe will know nothing about any of this, all he'll know is that the guy at Best Buy reliably informed him that the S4 smoked everything in its path when magazines benchmarked it, and Candy Crush will therefore look AWESOME!!! Job done, Samsung.

 

[Adam Starkey]

Sixclaws[/url]":2i5v92qc]The sad part is that the blame is going to land on the scapegoats from their R&D/engineering departments. The idiots from Upper Management who ordered this act of stupidity on the other hand are going to get away with it.

Samsung is not in the habit of eating crow. My guess is that no-one's going to be made a scape-goat for anything.

As I said before, Samsung is a company filled with some of the most amazing engineering and design teams the world has ever seen. Sadly, all that smart and creativity goes down the drain because of the idiot bosses.

Accusing Samsung's bosses of being idiots is seriously misguided.

 

[Adam Starkey]

aiken_d[/url]":2jzsagrr]The funny thing about this is that it probably didn't do them any good at all. iOS users prioritize UX over benchmarks. WP8 users work for Microsoft.

And Best Buy customers just want something to help them feel good about an arbitrary choice.

This seems to need repeating in almost every thread here, so I'll step up this time: you and I are reading Ars, we do not represent the broader market.

 

[Adam Starkey]

truepusk[/url]":3ul2dcfl]
Best Buy probably isn't the broader market anymore. I don't think many analysts think it will be long before they go the way of Circuit City and those who don't are probably betting on Best Buy to be able to make major changes to adapt.

No disagreement there. I was using BestBuy purely as a stand-in for generic gizmo shopping. The experience would be broadly the same at a Verizon store.

If/when BestBuy keels over, their customers will simply move to another store where they will continue to be only loosely informed about the devices they are purchasing, and only marginally more interested. The majority of the population really doesn't know or care much about their phones and tablets. They know they don't want to buy the wrong thing, and are only too happy to buy what everyone else is buying because there's safety in numbers. Of the small chunk of the market that is engaged with their purchases, a decent proportion of that group is really running on received wisdom. They don't know or understand half as much as they think they do, and their gut is doing as much thinking as their head. The remaining subset of the market makes what could be properly regarded as an informed choice.

As someone mentioned above, there is power in influencing the latter two groups because they (we) can have some effect on the larger market, but not half as much as a decent price (preferably one that appears to be a discount), and a recognized brand name.

This is of course as true of TVs, washing machines, cars, and power tools, as it is of phones, tablets, or laptops. None of us have enough time to *really* know what we're doing when we make half the purchases we make, and only some of us are interested enough in any of those things to make the time.

I doubt this will hurt Samsung any in the short term. In the long term it may begin to coalesce into a broader narrative but that'll take a while, and in the interim Samsung is making money hand over fist.

 

[Adam Starkey]

therealankit[/url]":2v5kujsw]
Also, if some benchmarking applications make the device clock higher, isn't it possible if developers make applications that makes the device faster all the time, to get the best usage. That will be good for the users.

No it wont. Samsung aren't throttling the performance of their hardware for the lulz. Their testing obviously led them to set an upper safe/functional bound on their devices. Lots of apps a) messing around masquerading as other apps, and b) forcing a device to run out of specification for protracted periods of time would be all round bad.

At best it's a monumental waste of developer time that only impacts some Android devices, at a cost of significant battery life. At worst, it's a whole lot of users dealing with crashes, reboots, and shorted device life caused by overheating.

 

[Adam Starkey]

f0xik[/url]"04w5gcw]
nVidia lists in their driver release notes for which games they improved the performance. You can also edit the application profiles through their control panel. I believe that's transparent enough.

Tyler X. Durden already explained why what nVidia/ATI were doing was not just tweaking performance for popular games.