Samsung defends benchmark boosting because it boosts other apps, too

But the fact remains: benchmarks get boosted, most games don't.

Read the whole story

 

[redleader]

Finally, a point of clarification here: as of right now, the international version of the Galaxy S 4 with Samsung's Exynos 5 Octa system-on-a-chip (SoC) is the only one that's doing this. The Snapdragon-equipped US version of the phone that many of our North American readers will be using isn't exhibiting the same behavior.

Its no secret that the Exynos 5 runs hotter than intended and with less than expected performance due to a series of design errata that forced Samsung to turn to their rivals at Qualcomm for the new S4 in many regions. I think its pretty obvious that what happened here is that they decided to try and make it look like their part was a little more competitive in order to save face and hopefully look more competent when the fixed Exynos 5 hardware ships later this year or early next.

Hence, they boosted the E5 benchmarks in order to narrow the perf/battery life gap with the Qualcomm model.

 

[Lonyo]

I'm sure some people might buy a phone based on benchmarks, but to me there are far more important things than outright performance, that even a 10% improvement in benchmarks wouldn't outweigh.

There's a lot more differentiation in phones these days than in PCs back in the day in many areas, that you can't change through upgrades as an end user.

I bought a Galaxy S3 last year. Didn't even look at benchmarks. It was the only high end phone with a removable battery and MicroSD card slot, so there were no other choices.

 

[Stuka87]

It does help some of their own apps. But the fact that they specifically have code that bumps it up for benchmarks is still at issue.

If games have to run at a lower clock to keep the thing from spontaneously combusting, than the benchmarks should as well to reflect real world performance.

 

[BigLan]

The throttling sounds like they were trying to do something like Nvidia's GPU Boost, which allows for their chips to run faster than usual as long as overall power/heat thresholds aren't crossed. The difference is that nvidia (hopefully) does that across the board by using actual power/heat measurements, and not just by detecting the application.

I think it was mentioned in the comments on the earlier story, but AMD was caught taking that approach back with quake3 (renaming it to quack3.exe gave lower performance.)

 

[Mazzicc]

I'd actually just prefer that they offer benchmarking apps the ability to run at both speeds, so you can see how the apps that use 533MHz mark, and how the games (is it only games?) at 480MHz mark.

 

[Stuka87]

Mazzicc[/url]":32p4avv8]I'd actually just prefer that they offer benchmarking apps the ability to run at both speeds, so you can see how the apps that use 533MHz mark, and how the games (is it only games?) at 480MHz mark.

Samsung says it is full screen apps that use the GPU. Except for some of their built in Apps that they specifically tell to run at the higher speed, like the camera.

 

[thomsirveaux]

Lonyo[/url]":201xhhjp]I'm sure some people might buy a phone based on benchmarks, but to me there are far more important things than outright performance, that even a 10% improvement in benchmarks wouldn't outweigh.

There's a lot more differentiation in phones these days than in PCs back in the day in many areas, that you can't change through upgrades as an end user.

I bought a Galaxy S3 last year. Didn't even look at benchmarks. It was the only high end phone with a removable battery and MicroSD card slot, so there were no other choices.

IMO it's a bit less about "people will buy this phone because it has better benchmarks" and more about Samsung not being honest with people.

 

[grahamwilliams]

The fact of the matter is that Samsung is comfortable with fudging the numbers to come out on top. This isn't even a reality distortion field thing, it was an active effort to lie to consumers.

Whether you buy a Samsung phone is up to you, but I know I'm sticking with LG, Sony, and HTC for Android from here on out.

 

[Chuckstar]

doppio[/url]":3r8zjc7n]I for one find the explanation perfectly reasonable. Clock frequency is being modulated based on type of apps running. It would be stupid not to.

High-GPU-utilizing app that is a game: use lower clock speed
High-GPU-utiltizing app that is a benchmark: use highest clock speed

How is that perfectly reasonable?

 

[passivesmoking]

If reviewers stop publishing benchmarks there'll be lest incentive to cheat like this.

Not trying to justify what Samsung did, because it's obviously utterly scummy. Just sayin'.

 

[Leiesoldat]

necrosis[/url]":1z556lyx]I just find it more interesting that people actually benchmark phones.

Err, you're just now finding this out?

 

[deviladv]

doppio[/url]":1crqme60]I for one find the explanation perfectly reasonable. Clock frequency is being modulated based on type of apps running. It would be stupid not to.

However the devil is in the details.

Samsung is not boosting games to 533 Mhz, like they said.
Samsung is only boosting a couple of baseline apps, like the camera, and benchmarking apps.

The "scandal" is equivalent to advertising a sports car that can go 220 MPH, but the only time it seems to go that fast is when a reviewer from a magazine is sitting in the passenger seat. All other times it goes 180. It's not a killer problem for the public at large, since speed limits are generally much lower than this in most places other than the German Autobahn. But if you were one of the rare nerds who is concerned about MHz on a phone and you bought the phone for that reason, being truthful matters to you. This isn't an earth shattering scandal, but it does cross the line from truth to lying. If Samsung continues to do things like this, we'll harken back to this situation as a possible marker for a pattern. If Samsung never does it again, it's a blip.

 

[Drakkenmensch]

If they did this to avoid undue stress on the hardware (a reasonable concern), then why wait until the accusations of benchmark gaming to surface before letting us know of this technique?

In the realm of public opinion, the cover-up is always far, far worse than whatever it is that you tried to cover.

 

[TheFerenc]

BigLan[/url]":xlsykowk]I think it was mentioned in the comments on the earlier story, but AMD was caught taking that approach back with quake3 (renaming it to quack3.exe gave lower performance.)

Nitpick: It was ATi back then, not AMD. Quake 3 was released in 1999. AMD didn't aquire ATi until what, 2006?

Big difference, there.

 

[Mydrrin]

Lonyo[/url]":1vvz63ri]I'm sure some people might buy a phone based on benchmarks, but to me there are far more important things than outright performance, that even a 10% improvement in benchmarks wouldn't outweigh.

There's a lot more differentiation in phones these days than in PCs back in the day in many areas, that you can't change through upgrades as an end user.

I bought a Galaxy S3 last year. Didn't even look at benchmarks. It was the only high end phone with a removable battery and MicroSD card slot, so there were no other choices.

Many people buy phones off of the recommendations of sites like Ars or Anandtech, if they say the best bang for the buck is the S4 or the One. 10% faster is something, some might think would be a no brainer. Does everything the One does and is 10% faster. It really does make a big difference to recommendations from sites.

 

[obzbdc]

Chuckstar[/url]":16m08zdb]

doppio[/url]":16m08zdb]I for one find the explanation perfectly reasonable. Clock frequency is being modulated based on type of apps running. It would be stupid not to.

High-GPU-utilizing app that is a game: use lower clock speed
High-GPU-utiltizing app that is a benchmark: use highest clock speed

How is that perfectly reasonable?

Stab in the dark here, average running time? Probably a couple of minutes at most for the benchmark app and potentially much much longer for the game?

 

[21mhz]

Of course benchmarking apps demand the redline performance. Isn't their purpose to make the device look better and thus help sell it to people who buy devices based on dumb specs? The people who bought Pentium 4 space heaters are getting older (and wiser), but you know, a sucker is born every minute.

 

[Mydrrin]

deviladv[/url]":3nep8dta]

doppio[/url]":3nep8dta]I for one find the explanation perfectly reasonable. Clock frequency is being modulated based on type of apps running. It would be stupid not to.

However the devil is in the details.

Samsung is not boosting games to 533 Mhz, like they said.
Samsung is only boosting a couple of baseline apps, like the camera, and benchmarking apps.

The "scandal" is equivalent to advertising a sports car that can go 220 MPH, but the only time it seems to go that fast is when a reviewer from a magazine is sitting in the passenger seat. All other times it goes 180. It's not a killer problem for the public at large, since speed limits are generally much lower than this in most places other than the German Autobahn. But if you were one of the rare nerds who is concerned about MHz on a phone and you bought the phone for that reason, being truthful matters to you. This isn't an earth shattering scandal, but it does cross the line from truth to lying. If Samsung continues to do things like this, we'll harken back to this situation as a possible marker for a pattern. If Samsung never does it again, it's a blip.

I would say that it is more like the car company giving a suped up version of the car with a bigger engine displacement for testing how fast from 0-60. But when you guy the car instead of going 0-60 in 6.0 seconds it does it in 7.0.

I would term this more like the milage numbers from Kia and Hyundai. They said they could do 40mpg but no one could find a way to get to that number and generally found to do 36. They were trusted to do the proper tested not rig the testing. This is even in worse in my books, it is complete deception. For Hyundai and Kia they were forced to give vouchers for fuel. But the advertising that 40mpg got them lots of sales. They should have been punished far worse for deception, but how does one prove. This will get slammed hard, this is hand in the cookie jar caught, there is no oops we didn't calculate properly the milage - one would have to have irrefutable proof. This the proof is in the coding.

 

[mpat]

doppio[/url]":31i1ldch]

deviladv[/url]":31i1ldch]

doppio[/url]":31i1ldch]I for one find the explanation perfectly reasonable. Clock frequency is being modulated based on type of apps running. It would be stupid not to.

However the devil is in the details.

Samsung is not boosting games to 533 Mhz, like they said.

Source needed.

Other than Samsung saying as much, the Anandtech articles that started all of this, for instance.

http://www.anandtech.com/show/7187/look ... galaxy-s-4
http://anandtech.com/show/7192/update-o ... galaxy-s-4

 

[Chuckstar]

doppio[/url]":3g31d49g]

Chuckstar[/url]":3g31d49g]

doppio[/url]":3g31d49g]I for one find the explanation perfectly reasonable. Clock frequency is being modulated based on type of apps running. It would be stupid not to.

High-GPU-utilizing app that is a game: use lower clock speed
High-GPU-utiltizing app that is a benchmark: use highest clock speed

How is that perfectly reasonable?

The UI uses the higher clocking speed. Most people complain from stutter in the UI. Games usually run OK. Imagine the same game on a slower device. A few % down in the framerate will not be noticed.

What does the UI have to do with hard-coding certain benchmark apps to run at higher clock speed than any other app runs at?

 

[Flashlight]

doppio[/url]":2b7q5crv]

thomsirveaux[/url]":2b7q5crv]

Lonyo[/url]":2b7q5crv]I'm sure some people might buy a phone based on benchmarks, but to me there are far more important things than outright performance, that even a 10% improvement in benchmarks wouldn't outweigh.

There's a lot more differentiation in phones these days than in PCs back in the day in many areas, that you can't change through upgrades as an end user.

I bought a Galaxy S3 last year. Didn't even look at benchmarks. It was the only high end phone with a removable battery and MicroSD card slot, so there were no other choices.

IMO it's a bit less about "people will buy this phone because it has better benchmarks" and more about Samsung not being honest with people.

I don't believe Samsung advertise the benchmarks, so how are they dishonest?

The reaction on tech blogs is easy to understand -- tech writers have created a straw man -- benchmarks -- and are now upset that it doesn't behave as the actual thing.

How stupid many who published articles based around benchmarking must feel!

Don't become a lawyer or go into PR.

 

[Infinity4011]

I'd like to see some further investigation to see if other companies are similarly engineering their phones to abuse benchmarks.

At the same time, the outrage over it is chuckle worthy for a variety of reasons. For starters, tweaking one's product for a synthetic test has been and will continue to be a time-honored tradition. Car companies demo cars with better tires than the same product that they sell to the public at large; broadband companies advert speeds that are far higher than what you get, and companies like Google tweak their browers to score better on benchmarks.

Second, using synthetic benchmarks as a determination of performance is like tasting three of the many ingredients that go into a dish, before you cook it. You can infer some indicator of the performance of a product from synthetic benchmarks, but in a lot of cases high benchmarks can still leave a user wondering why their device "feels" slow, because of poor optimization or sloppy coding outside of the benchmark.

Third, synthetic benchmarks themselves are sometimes written (though not necessarily purposefully) to lean towards favoring certain browsers, GPUs, software, or hardware. Case in point, when reviewers of graphics cards use a stable of nVidia optimized games to compare AMD and nVidia cards, performance results are bound to be skewed. Or supporting hardware that's not as compatible (it might function, but not as well).

We would all like for everyone to just be honest, but with millions riding on one's products, it's a pipedream.

 

[jxmzsr]

For cripes sake people, when are you going to understand that companies only sell you a product so they can make money and could really care less that something runs slower on game apps. Of course it ran faster on benchmarks, "in the know" web sites benchmarked it, company mission to attract the nerd/tech crowd to buy it completed. They don't really care that because something runs better with benchmarks used to attract customers, if you like it or not.

 

[NickN]

Samsung essentially argues ... that the 533MHz speed is supported by a variety of apps other than benchmarks

Yeah, like for example their "BenchmarkBooster" app. I'm sure the name is a total coincidence.

 

[amvakar]

agenius[/url]":1dzkj5r5]you know, when I go to a job interview, I dont go in my PJs, with my bed hair just because this is somehow representative.

if it bothers you that much, think of it as performance art: Samsung are like, "well if you put us on the spot, we'll turn the tables & embarrass you!"

plus. nowhere on any of Samsung's pr or even their official spec do they say ANYTHING about the GPU's frequency

They're not turning the tables in any sense. It'd be more like going to a job interview dressed up nice, and then upon being told to dress up for some company function a week later having to say "well, yeah, I can't afford to rent that suit again." The only people who could come out of that exchange thinking themselves superior are people I'd suspect to have a mental disorder.

And while they may not be saying anything about performance, by allowing certain benchmarks to mislead the user about average performance, they're speaking implicitly. If it can't at least theoretically reach those levels all the time, they're lying bastards and there is no excuse.

 

[taswyn]

agenius[/url]":izb60iys]you know, when I go to a job interview, I dont go in my PJs, with my bed hair just because this is somehow representative.

if it bothers you that much, think of it as performance art: Samsung are like, "well if you put us on the spot, we'll turn the tables & embarrass you!"

plus. nowhere on any of Samsung's pr or even their official spec do they say ANYTHING about the GPU's frequency

...

So you go into the interview and put down a doctored resume? Throw around a lot of jargon you actually don't have a clue about?

When you're caught, you'll just say you were brushing up on your creative writing and speaking abilities? When you're caught you'll actually expect to get a free pass and be congratulated on your effort above and beyond to make a good impression?

Sorry, but why are you surprised over the outcry over an act that was clearly meant to be deceptive--and they got caught red handed? There was obvious intent to deceive here. I don't even get why it's in question, or why anyone is even attempting to defend it in this manner. The same goes for whether or not they advertised the CPU speed: it's immaterial. The point is they obviously targeted benchmarks for changing performance, which by itself is intentionally deceptive when the applications the benchmarks are supposed to mime in terms of performance do not receive the same performance.


A discussion on how various industries attempt to dupe consumers and falsify how they present their products to consumers is fine and relevant. Even questioning where this sits in that context is fine. But pretending Samsung didn't do anything wrong or duplicitous here is... it strains credulity. Yes, companies regularly try to pretend they're selling something better than what they are. That doesn't make it ok, or any less intentionally deceptive. "Everyone else is doing it" is not an ethical measure for a reason.

 

[Number_One_Fandroid]

The only problem here is that they should allow the user to choose whether and when they want to go into overdrive. If I want to overclock my phone in exchange for some additional heat and battery drain, why shouldn't I be able to?

 

[orangpelupa]

This is different from ati optimising for quake3, crysis 3, or mass effect 3.

It's like (if) ati did not show gpu overdrive to user and only enable gpu overdrive when running benchmark.

But there's still a difference. On ati, they advertise the gph clock. On Samsung, they did not advertise the gpu clock.

 

[doppio]

Number_One_Fandroid[/url]":2umucnbc]The only problem here is that they should allow the user to choose whether and when they want to go into overdrive. If I want to overclock my phone in exchange for some additional heat and battery drain, why shouldn't I be able to?

Great idea.

I want to be able to choose clock frequency, voltage, and other parameters of the hardware.

I also want to buy the device pre-rooted, unocked, etc.

Now, let's think how many end users will cry out that this is getting too complicated for them.

 

[gdinero]

Whether Samsung ever advertised GPU clocks is immaterial. They knew the phones were going to be benchmarked and added code to give benchmarks a performance advantage that the vast majority of apps running on the phone don't have.