Samsung allegedly boosting benchmark performance

Anandtech digs deep on a forum allegation and finds depressing results.

Read the whole story

 

[hobgoblin]

Yeah, i noticed that issue recently on a Archos device. Their media player seems to poke the storage space ever so often, making the whole device unresponsive for the duration. Disabled said app, and have not noticed any unresponsiveness since. This could be related to various issues Linux has had in the past with storage operations making GUI interactions unresponsive, leading to all kinds of scheduler shenanigans and flared tempers on the LKML.

 

[Walt French]

atomo[/url]":2aoj26p3]

MacsAre1[/url]":2aoj26p3]Two words: Battery life.

One word: Marketing.

Let me stand up and say that good marketing never takes the kind of shortcut that gets you quick sales and then tells customers that you're exploiting them. It *IS* possible.

Along the lines of Gresham's Law, accepting deception in place of marketing guarantees that everybody lies to you, by making it impossible for a company to get noticed without making hyped claims. The shock here is that Samsung broke out of the convention of honesty; they've invited more government regulation by breaking the code of honesty that allowed the deception to succeed.

 

[Walt French]

TheDissolver[/url]":13e3vvcj]the benchmark means fuck-all to user experience. so I give precisely one tenth of a shit…

Exactly, this is all about marketing, where Samsung shapes your impressions about how much better their phone is than others' devices.

About why you should buy it instead of an HTC One, for example, an either-or choice that many people will make. HTC, which is on the edge of profitability, gets put into a bad place by this type of propaganda: either they dope too, or have to drop out of the race. Not caring means letting deception winnow out the ranks of competition for your purchases, supposedly a principle feature/advantage of the Android ecosystem.

 

[Walt French]

MrMickS[/url]":1hwok1nk]Perhaps its time that tech web sites, such as Ars, stopped using benchmarks in their phone reviews. They could include a short paragraph to say why.

I don't care about the benchmark performance of my phone. I do care how responsive the UI is, how quickly it can render web pages, how it performs in a low signal area. What happens when it can't get that WiFi or LTE connection. Instead of reviewing a smartphone as a computer, come up with a series of tests that test its ability as a smartphone.

The whole reason that Samsung gamed the benchmarks is that previously, they were good measures of how fluid and fast a given function, e.g., gaming, was on a given device. If a game uses OpenGL, then OpenGL benchmark scores across different phones ought to give a good signal as to whether, all else equal, you'd be happier on the Galaxy or the One.

Your proposal throws out the baby with the bathwater, and would reduce reviewing to the sort of subjective blathering that is even MORE susceptible to marketing spin. Versus, reviewers demand the company to certify that review units have not been altered to make them look better than what ordinary users will experience. The latter is easy and honest companies won't have any trouble putting their signatures on the line—especially if Ars or others put a little note saying, “the company wouldn't attest that our benchmark suites are indicative of what users should expect for similar functions.”

 

[chocoruacal]

Mobile benchmarks don't mean jack s***. Proof: the Galaxy S3, which had phenomenal benchmarks (at the time, obviously) but which lags like something out of 2008.

 

[Tundro Walker]

So it's the tech-version of "No Child Left Behind". Spend energy "teaching to the test", which means less energy (and focus) is spent on the primary goal. (In this case end-user experience instead of benchmark performance).

 

[1337 poster]

Marlor[/url]":3niv0bih]Their fridges could somehow detect when they were in the energy efficiency testing lab, and throttle their cooling to achieve impossibly impressive results.

I would love to know how this worked.

 

[donelson]

ShlomoAbraham[/url]":2kh8k7g6]Any theories as to why? If the chip is there, why not let other apps use it? Save battery life?

Heat and battery life.

 

[Grimmash]

SedsAtArs[/url]":up3agbsn]

Grimmash[/url]":up3agbsn]

deathBOB[/url]":up3agbsn]Boy I sure don't care about mobile benchmarks.

To play devil's advocate, mobile computing is still an area where performance is still something that could be a factor, depending on application. For desktops, most people can afford a computer that far exceeds the demands of actually critical software. While Ars might be a hotbed of people tweaking to get Crysis or some similar application running at 120 FPS or something, mobile devices can still fail to provide the proper sweet spot of speed and battery life. I know the Macbook Air I am typing this on is woefully unable to do many things my desktop, a full two years older, can just shrug off.

Cheating mobile benchmarking could have actual consequences when mobile phones can cost as much as a video card with contract, and a video card + ram + something else without contract.

While I agree what he posted won't be true for all users, and that it's a shady practice by Samsung in any case, it could still very well be true for the poster you quoted.

I know that myself, I mostly use very simple apps, which don't tax my phone much. Google Maps and Ingress are probably the two slowest running ones, and I don't feel I can fully blame the slowness of Ingress on my phone.
The day that docking a phone in and using it as one does a laptop for work now comes (assuming it does), this might well change for me.

Fair enough. My point was that there are companies that may actually care about every bit of phone performance. If you are setting up a big contract for all your employees, and expect them to have a phone that replaces a lot of computer functionality (say for highly mobile field reps), fake benchmarking could be an issue and result in a non trivial financial consideration. So even if your personal use case could care less, I was trying to illustrate who this may be worthwhile news for some people.

My initial response was the same as the post I quoted, but then I thought about non-personal phone uses.

 

[21mhz]

MrMickS[/url]":1g6eubor]I don't care about the benchmark performance of my phone. I do care how responsive the UI is, how quickly it can render web pages, how it performs in a low signal area. What happens when it can't get that WiFi or LTE connection. Instead of reviewing a smartphone as a computer, come up with a series of tests that test its ability as a smartphone.

But then, how would we bash Windows Phone for lacking the pointless specs "from 2013"?

 

[Shudder]

I guess Samsung felt the need to do this because HTC was getting a lot of good press with their performance (same chip as the S4, so it was good) and the superior screen and build quality.

It's disheartening to see the market leaders freak out and do things like this because shareholders will freak out that their growth was only x-1% instead of x%.

Saddest part is the phone would have done fine on its own thanks to their 827 million dollar (estimated) ad spend to push it. Cheating was just a stupid way to try to get ahead when they didn't have anything to positively separate themselves from the other guy.

 

[Walt French]

Batmanuel[/url]":32lbb20i]Is this tweak valid on the US models? Those use a quad-core Qualcomm chip with a totally different GPU. Has anyone checked them for similar code?

These may be “international” models but they are approved for sale in the US and certified as complying with US regulations.

Naturally, manufacturers will tweak individual models for particular markets. It may be that the same firm that organized astroturfers in Taiwan, and doped their benchmark results, otherwise conducts its marketing with utter and complete integrity. While you're quite correct in that individual models might not have this deception coded against them, Samsung has opened up the can of worms that ALL their models now need to be scrutinized for possible variances between test and actual performance. We say “marketing,” but this is obviously the handiwork of a team of engineers working under a top-down directive, the team that'd be involved in a whole host of products.

The thing *I* don't get is who in their right mind would've authorized and organized this. The chance of it being exposed seems fairly high; actual fines or sanctions seem possible in some places (such as Taiwan, where Sammy was held responsible for non-employees' claims); there is the risk of incredible harm to the Samsung brand. And until the music stops, the fake performance boost is so modest that it'd boost sales not terribly much. How does a corporate culture encourage trashing their reputation so cheap?

 

[Stuka87]

Firehawke[/url]":3amztalo]ATi and nVidia were both caught in the past doing this with video card benchmarks, too. I particularly remember one incident involving Quake 3 benchmark optimizations about twelve years ago where ATi was doing some specific optimizations that would only trigger when it detected the calling application was "quake3.exe"

Made it really easy to prove, too. Just rename the EXE and you'd see performance drop considerably.

This is not the same thing. Optimizing games is what GPU makers do. This HELPS the end customer. And game optimizations are almost always done by the EXE name. This holds true to today. If nVidia or AMD have some optimizations for say The Witcher 2, when that exe is launched, their optimizations kick in. Driver optimizations are perfectly normal and accepted as normal practice.

What Samsung is doing is only boosting for benchmarks. People do not use benchmarks in their day to day life. Purposely over clocking for these is wrong, period.

 

[smokedart]

chocoruacal[/url]":2jafhphs]Mobile benchmarks don't mean jack s***. Proof: the Galaxy S3, which had phenomenal benchmarks (at the time, obviously) but which lags like something out of 2008.

Or maybe Samsung was gaming the benchmarks back then also?

 

[Me Myself And I1]

Some car makers map the engines specifically to suit mileage testing standards, some design crash zones specifically to perform well at standardized crash tests to the detriment of real life situations.

I think that's cheating the consumer and same applies to Samsung benchmark tweaks.

It's marketing, it's buzz, it's the dumb public (and general/generic press) that only listens to bigger's better. So you ramp up GHz, you get more stars at crash tests, you get better mileage numbers and you increase number of megapixels - all on paper.

Once you can no longer push those figures, you begin being creative with these sort of things - it's cheaper and easier than actually innovate.

 

[FlyingKiwi]

atomo[/url]":2tmd2jq8]This is like selling a car that you say has 500hp but only when you're running it on the Bonneville Speedway. Drive it anywhere else and the hp drops down to 400.

It's all for marketing and bragging rights. This is dishonest at best and downright deceptive and possibly illegal at worse.

It's more like a car that has a governor in it. Say the car can go to 200mph but it is restricted to 150mph by the governor. Then you see the car run at an auto race where it is going the full 200mph. You can still have the car go to 200mph if you remove the governor (read: root/unlock the device and modify the clocks) but for the average person 150mph is good enough and the car won't have nearly as much wear-and-tear on it compared to if it could go up to 200mph (In a phone the same concept applies a CPU/GPU that's overclocked would likely not last as long than a stock clocked CPU/GPU). I agree that a little more truth in advertising might be good from Samsung but showing the full potential of a CPU/GPU to a benchmarking application isn't that surprising.

 

[Drakkenmensch]

FlyingKiwi[/url]":3ccxqmvu]

atomo[/url]":3ccxqmvu]This is like selling a car that you say has 500hp but only when you're running it on the Bonneville Speedway. Drive it anywhere else and the hp drops down to 400.

It's all for marketing and bragging rights. This is dishonest at best and downright deceptive and possibly illegal at worse.

It's more like a car that has a governor in it. Say the car can go to 200mph but it is restricted to 150mph by the governor. Then you see the car run at an auto race where it is going the full 200mph. You can still have the car go to 200mph if you remove the governor (read: root/unlock the device and modify the clocks) but for the average person 150mph is good enough and the car won't have nearly as much wear-and-tear on it compared to if it could go up to 200mph (In a phone the same concept applies a CPU/GPU that's overclocked would likely not last as long than a stock clocked CPU/GPU). I agree that a little more truth in advertising might be good from Samsung but showing the full potential of a CPU/GPU to a benchmarking application isn't that surprising.

To add on to your analogy, the manufacturer of that car is making sure to provide a test model to the car reviewer that will disable the speed blocker when the car is rolling on a test drive range, giving the reviewer the illusion that it can go at 200mph by default...

 

[aardman]

steven75[/url]":15l99v41]Samsung: Truly a company to be admired for its morals and ethics.

Why would you say that? You're prejudging the whole company's morals and ethics just because the CEO has been convicted of tax evasion and fraud. Unfair!

/s (for those missing the irony gene)

 

[FlyingKiwi]

Drakkenmensch[/url]":34szclu8]

FlyingKiwi[/url]":34szclu8]

atomo[/url]":34szclu8]This is like selling a car that you say has 500hp but only when you're running it on the Bonneville Speedway. Drive it anywhere else and the hp drops down to 400.

It's all for marketing and bragging rights. This is dishonest at best and downright deceptive and possibly illegal at worse.

It's more like a car that has a governor in it. Say the car can go to 200mph but it is restricted to 150mph by the governor. Then you see the car run at an auto race where it is going the full 200mph. You can still have the car go to 200mph if you remove the governor (read: root/unlock the device and modify the clocks) but for the average person 150mph is good enough and the car won't have nearly as much wear-and-tear on it compared to if it could go up to 200mph (In a phone the same concept applies a CPU/GPU that's overclocked would likely not last as long than a stock clocked CPU/GPU). I agree that a little more truth in advertising might be good from Samsung but showing the full potential of a CPU/GPU to a benchmarking application isn't that surprising.

To add on to your analogy, the manufacturer of that car is making sure to provide a test model to the car reviewer that will disable the speed blocker when the car is rolling on a test drive range, giving the reviewer the illusion that it can go at 200mph by default...

That's a good point and again Samsung should be letting people know about them doing this. Perhaps they could have made some way for the benchmarks run off either the clock speeds or the limited clock speed imposed on other apps (does anyone know if it exposes the full clock speed to the Android OS or not? That would be interesting). That way people would know if they wanted to remove the limitor and risk the extra preformance to battery life / wear-and-tear or stay at the limited clock.

 

[nugundam93]

atomo[/url]":1an76e02]This is like selling a car that you say has 500hp but only when you're running it on the Bonneville Speedway. Drive it anywhere else and the hp drops down to 400.

It's all for marketing and bragging rights. This is dishonest at best and downright deceptive and possibly illegal at worse.

just like a JDM GT-R which is speed-limited until its GPS detects you're at one of the Nissan-recognized racetracks and then the gloves are off.

i'm honestly quite ok with this - while i know the best that my phone can do, i also do want it to work as well when i need it, not just during a pissing contest. they should have an unlock setting or something with a disclaimer that while it'll give you the phone's full potential, be prepared for shorter component and battery life.

 

[FlyingKiwi]

Me, Myself And I[/url]":2z2e7peq]Some car makers map the engines specifically to suit mileage testing standards, some design crash zones specifically to perform well at standardized crash tests to the detriment of real life situations.

I think that's cheating the consumer and same applies to Samsung benchmark tweaks.

It's marketing, it's buzz, it's the dumb public (and general/generic press) that only listens to bigger's better. So you ramp up GHz, you get more stars at crash tests, you get better mileage numbers and you increase number of megapixels - all on paper.

Once you can no longer push those figures, you begin being creative with these sort of things - it's cheaper and easier than actually innovate.

Don't forget giving game reviewers early copies in exchange for high scores on the game.

It's become a "Who is better at reviews" world. I still try and listen more to word of mouth, friends, and user reviews (grain of salt to that one) and actually try to test out the device for myself before I buy over what reviewers say (sorry Ars this even applies to your reviews). It's becoming harder to do some of those things however.

 

[Drakkenmensch]

FlyingKiwi[/url]":10tcydzn]

Drakkenmensch[/url]":10tcydzn]

FlyingKiwi[/url]":10tcydzn]

atomo[/url]":10tcydzn]This is like selling a car that you say has 500hp but only when you're running it on the Bonneville Speedway. Drive it anywhere else and the hp drops down to 400.

It's all for marketing and bragging rights. This is dishonest at best and downright deceptive and possibly illegal at worse.

It's more like a car that has a governor in it. Say the car can go to 200mph but it is restricted to 150mph by the governor. Then you see the car run at an auto race where it is going the full 200mph. You can still have the car go to 200mph if you remove the governor (read: root/unlock the device and modify the clocks) but for the average person 150mph is good enough and the car won't have nearly as much wear-and-tear on it compared to if it could go up to 200mph (In a phone the same concept applies a CPU/GPU that's overclocked would likely not last as long than a stock clocked CPU/GPU). I agree that a little more truth in advertising might be good from Samsung but showing the full potential of a CPU/GPU to a benchmarking application isn't that surprising.

To add on to your analogy, the manufacturer of that car is making sure to provide a test model to the car reviewer that will disable the speed blocker when the car is rolling on a test drive range, giving the reviewer the illusion that it can go at 200mph by default...

That's a good point and again Samsung should be letting people know about them doing this. Perhaps they could have made some way for the benchmarks run off either the clock speeds or the limited clock speed imposed on other apps (does anyone know if it exposes the full clock speed to the Android OS or not? That would be interesting). That way people would know if they wanted to remove the limitor and risk the extra preformance to battery life / wear-and-tear or stay at the limited clock.

I can understand the reasoning behind the idea of limiting the CPU performance in order to preserve the
battery and prevent heat or wear related issues, but it's the deception behind everything that I'm objecting to, giving consumers the illusion that it can go that fast when in truth the system is blocked from achieving the level of performance that benchmarks are shown to be capable.

 

[Tundro Walker]

Shudder[/url]":1ykrw4hc]I guess Samsung felt the need to do this because HTC was getting a lot of good press with their performance (same chip as the S4, so it was good) and the superior screen and build quality.

It's disheartening to see the market leaders freak out and do things like this because shareholders will freak out that their growth was only x-1% instead of x%.

Saddest part is the phone would have done fine on its own thanks to their 827 million dollar (estimated) ad spend to push it. Cheating was just a stupid way to try to get ahead when they didn't have anything to positively separate themselves from the other guy.

You're right about the investors part. Blackberry reported less-than-expected earnings. Their stock went from $14/share to $9/share. ($8/share last I checked).

Investors wouldn't screw with the market so much if "shorting" (making a profit on a stock going down) was no longer allowed.

 

[potato44829]

Well, at least it isn't Motorola, with their "8-core" chip.

My raspberry pi runs a tri-core 2.1GHz processor, then! Eat it up, iPhone!

 

[Penforhire]

To me the only part of this that is bogus is how Samsung doesn't expose that run-at-max capability to any other applications. If the reason is thermal then put a sensor in there and step it down when it gets warm. But don't keep something like this locked and in your pocket to use just for benchmarks.

 

[dehildum]

sd4f[/url]":2ju9nyry]
Michelin restaurant reviews have the right approach. Come in unannounced, pay cash and don't tell who they are. Once you put aside the reviewers inherent bias or slant (some people like things others don't), you can be sure that the review is a faithful experience.

Which is why most high end restaurants pay huge bonuses to staff who spot a reviewer. Pictures of well known food critics or food critics of large newspapers are posted in the kitchen so that staff can spot them. Same with the various other guides' reviewers. Unless the reviewer is in disguise, the meal, and service, they get is NOT what you will get.

 

[swarmster]

Samsung responded to the BBC:

http://www.bbc.co.uk/news/technology-23512887

"Under ordinary conditions, the Galaxy S4 has been designed to allow a maximum GPU frequency of 533MHz," it said.

"However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode," it said.

"Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player and certain benchmarking apps, which also demand substantial performance."

So it would seem perhaps a new headline is in order: "Samsung phones will overheat unless they release an update for every new gaming app"

Oh wait, their excuse is a seemingly proven lie and complete BS, nevermind.

 

[Number_One_Fandroid]

Eh, I like it. Samsung has demonstrated a killer instinct and a willingness to push the boundaries that other OEMs/platforms (Apple, Nokia/Microsoft, BB) have not. That's why they're the number one mobile platform in the world!

 

[Goofball_Jones]

Listen...you hear that? Listen closer...yes....can hear it clearly now. The sound of John Gruber pleasuring himself over this story.

 

[Eric72]

What's new? Samsung has made a habit of pulling the wool over consumers eyes.

 

[Sp3sm30]

Tundro Walker[/url]":21cblhgu]So it's the tech-version of "No Child Left Behind". Spend energy "teaching to the test", which means less energy (and focus) is spent on the primary goal. (In this case end-user experience instead of benchmark performance).

Not sure about your political reference but: Performance is part of the user experience. If it wasn't important they wouldn't care enough to cheat.

 

[Sp3sm30]

Number_One_Fandroid[/url]":2d984tql]Eh, I like it. Samsung has demonstrated a killer instinct and a willingness to push the boundaries that other OEMs/platforms (Apple, Nokia/Microsoft, BB) have not. That's why they're the number one mobile platform in the world!

Now sure why you are excited about them being number one if they got there by doing sh*t like this. As a customer do you enjoy being cheated and lied to much?

 

[For filtering out morons]

Violynne[/url]":3t15r1bf]I'm going to try and stave this issue by reducing the apps I install (so far, only 8), but to me, this seems to be counter-intuitive of using a *smart*phone.

[trollhide]I think you lack the qualifications to use any "Smart" phone, TV, toilet or anything else.[/trollhide]

 

[Number_One_Fandroid]

Sp3sm30[/url]":2zxw4aiv]

Now sure why you are excited about them being number one if they got there by doing sh*t like this. As a customer do you enjoy being cheated and lied to much?

If it sells more units, I'm not going to complain...

 

[potato44829]

For filtering out morons[/url]":g43xxwlx]

Violynne[/url]":g43xxwlx]I'm going to try and stave this issue by reducing the apps I install (so far, only 8), but to me, this seems to be counter-intuitive of using a *smart*phone.

I think you lack the qualifications to use any "Smart" phone, TV, toilet or anything else.

[trollhide]And you apparently lack the qualifications to post on Ars, one qualification of which would be to read the "no trolling" rule.[/trollhide]

/// OFFICIAL MODERATION NOTICE ///

---

No need to feed the troll or armchair moderate; please report trolls instead.

 

[Lord Sidious]

The arms race was started by pseudo tech journalism over-focusing on benchmarks...

 

[matthewslyman]

Walt French[/url]":2sgrzfjh]

matthewslyman[/url]":2sgrzfjh]Would this be grounds to retroactively reduce all of Samsung's recent history of benchmarking results by a factor of 533/480?

Following on their astroturf campaign, I think Samsung has now earned a “guilty until proven innocent” approach that your idea suggests...

Perhaps, but if we're going to continue criticizing Samsung for their astroturf campaign, shouldn't we also turn our attention toward Apple's "engineering" of their public-relations image? Their manipulations of the normal, healthy relationship between a tech company and the journalistic profession? I'm tempted to believe that in addition to banning journalists from Apple events after a negative review, they also pad out their events with screaming cheerleaders posing as journalists, bloggers and enthusiasts. Their products are fine, but I don't understand the hysteria they're trying to whip up around their brand and I'm getting tired of the aggressive cycle of carefully choreographed "announcements". So in the interests of balance, can we take all of these companies for what they are: profit-making machines with very little conscience besides that of the fallible people who direct and operate them?

 

[potato44829]

matthewslyman[/url]":1mwsc9vi]

Walt French[/url]":1mwsc9vi]

matthewslyman[/url]":1mwsc9vi]Would this be grounds to retroactively reduce all of Samsung's recent history of benchmarking results by a factor of 533/480?

Following on their astroturf campaign, I think Samsung has now earned a “guilty until proven innocent” approach that your idea suggests...

Perhaps, but if we're going to continue criticizing Samsung for their astroturf campaign, shouldn't we also turn our attention toward Apple's "engineering" of their public-relations image? Their manipulations of the normal, healthy relationship between a tech company and the journalistic profession? I'm tempted to believe that in addition to banning journalists from Apple events after a negative review, they also pad out their events with screaming cheerleaders posing as journalists, bloggers and enthusiasts. Their products are fine, but I don't understand the hysteria they're trying to whip up around their brand and I'm getting tired of the aggressive cycle of carefully choreographed "announcements". So in the interests of balance, can we take all of these companies for what they are: profit-making machines with very little conscience besides that of the fallible people who direct and operate them?

And Motorola too....

Hank, my world has collapsed! I can trust no one!

 

[NickN]

Number_One_Fandroid[/url]":2binbiwd]

Sp3sm30[/url]":2binbiwd]

Now sure why you are excited about them being number one if they got there by doing sh*t like this. As a customer do you enjoy being cheated and lied to much?

If it sells more units, I'm not going to complain...

Because you're making a commission on their sales?

 

[Metaluna]

Lord Sidious[/url]":1svrlrv4]The arms race was started by pseudo tech journalism over-focusing on benchmarks...

The worst are the ones that zoom in on the graphs. So if a test measures performance on a scale of 1-100, they'll only show the range from 95-100, to make it look like tiny performance differences between models matter.