Are benchmark tests a good representation of real life performance?

The performance of mobile devices has rapidly increased over the last few years. Major cell phone manufacturers are constantly trying to one-up each other more than ever before. It seems like every other day a new phone comes out with an even higher-resolution screen, a new-age GPU (Graphics Processing Unit), a ridiculously fast processor and an increasing amount of RAM (Random Access Memory). The mobile power race hasn’t gone unnoticed as consumers everywhere are hungry for beefier performance at the palm of their hands. Those curious to see how powerful their device truly is put it through a test called a benchmark.

The most common handsets that are put to the benchmark tests are those running on Google’s highly popular Android platform. There is an abundance of benchmarking applications available in the Android Market, in which the most notable is called Quadrant. What benchmarking does is measure multiple performance aspects of the device and add the results together to get a final score. The higher the score your smartphone receives, the better.

The main points measured in each test using one of the benchmarking applications are CPU, memory, I/O, and 2D/3D graphics. Powerful devices such as the Motorola XOOM tablet and the HTC Inspire 4G have performed well in these benchmark tests, but, interestingly, devices with similar specifications can often end up with an entirely different score altogether. So the question must be asked; can we trust these results to resemble real world performance? Or are these results skewed by other variables such as different screen resolution and user interfaces (custom software)?

As we saw years ago with the progression of PCs, the speed of the processor isn’t everything. Fortunately, manufacturers seem to be aware of this as evidenced by their advancements. There are many attributes that can affect the performance of a mobile device positively or negatively. A solid example is going back to the HTC EVO 3D benchmark that we did just a few weeks ago. While it may boast a 1.2GHz dual-core Qualcomm Snapdragon MSSM8660 processor to go along with a whopping 1GB of RAM and an Adreno 220 GPU, the EVO 3D was outperformed by other similar devices such as the Samsung Galaxy S II. But why?

One of the things that these tests do not take into consideration is the added user interfaces that manufacturers like to build on top of the Android platform. The HTC EVO 3D features the latest of HTC’s Sense UI which not only likes to eat away at your power but also occupies a good chunk of the device's memory. Given that memory is one of the main things measured by a benchmark test like Quadrant, it should be no surprise that the EVO 3D scores lower than the Galaxy SII which uses a somewhat less invasive user interface.

One of the great things about the Samsung Galaxy S II is the gorgeous Super AMOLED Plus technology of the display. However, this screen only has a resolution of 480x800. The HTC EVO 3D may not have the big, eye-popping colors of Samsung’s latest flagship, but it does however boast a sharp qHD screen with a higher resolution of 540x960. And the higher resolution display will take away from the EVO 3D’s score because it is taxing the processor more during usage.

So what can you take from all of this? Can we trust benchmark results to resemble real world performance? The answer can actually be both yes and no. First of all, we have to be sure that  there aren't any hardware/software incompatibilities between the device and the benchmarking app, which can be difficult to figure out, but suspiciously high or low results should lead to such thoughts. Then, it should be kept in mind that it's best to compare handsets with roughly the same system software installed, since it's obvious that custom solutions by manufacturers/developers may alter the benchmark results. That, however, doesn't mean that real-life performance will truly be different.

What do you all think? Are benchmark tests accurate means of testing the overall power of a device? Is it a good way to determine how well the phone will perform under real life conditions? We’d love to hear what you think on the subject so tell us what you think in the comments below!