3DMark for Android: Performance Preview
As I mentioned in our coverage of GL/DXBenchmark 2.7, with the arrival of Windows RT/8 we’d finally see our first truly cross-platform benchmarks. Kishonti was first out of the gate, although Futuremark was first to announce its cross platform benchmark simply called 3DMark.
Currently available for x86 Windows 8 machines, Futuremark has Android, iOS and Windows RT versions of 3DMark nearing release. Today the embargo lifts on the Android version of 3DMark, with iOS and Windows RT to follow shortly.
Similar to the situation with GL/DXBenchmark, 3DMark not only spans OSes but APIs as well. The Windows RT/8 versions use DirectX, while the Android and iOS versions use OpenGL ES 2.0. Of the three major tests in the new 3DMark, only Ice Storm is truly cross platform. Ice Storm uses OpenGL ES 2.0 on Android/iOS and Direct3D feature level 9_1 on Windows RT/8.
The Android UI is very functional and retains a very 3DMark feel. There’s an integrated results brower, history of results and some light device information as well:
There are two options for running Ice Storm: the default and extreme presets.
|3DMark – Ice Storm Settings|
Both benchmarks are rendered to an offscreen buffer at 720p/1080p and then scale up to the native resolution of the device being tested. This is a very similar approach we’ve seen by game developers to avoid rendering at native resolution on some of the ultra high resolution tablets. The beauty of 3DMark’s approach here is the fact that all results are comparable, regardless of a device’s native resolution. The downside is we don’t get a good idea of how some of the ultra high resolution tablets would behave with these workloads running at their native (> 1080p) resolutions.
Ice Storm is divided into two graphics tests and a physics test. The first graphics test is geometry heavy while the second test is more pixel shader intensive. The physics test, as you might guess, is CPU bound and multithreaded.
Before we get to the results, I should note that a number of devices wouldn’t complete the tests. The Intel based Motorola RAZR i wouldn’t run, the AT&T HTC One X (MSM8960) crashed before the final score was calculated so both of those devices were excluded. Thankfully we got the Galaxy S 3 to complete, giving us a good representative from the MSM8960/Adreno 225 camp. Thermal throttling is a concern when running 3DMark. You have to pay close attention to the thermal conditions of the device you’re testing. This is becoming something we’re having to pay an increasing amount of attention to in our reviews these days.
Graphics Test 1
Ice Storm Graphics test 1 stresses the hardware’s ability to process lots of vertices while keeping the pixel load relatively light. Hardware on this level may have dedicated capacity for separate vertex and pixel processing. Stressing both capacities individually reveals the hardware’s limitations in both aspects.
In an average frame, 530,000 vertices are processed leading to 180,000 triangles rasterized either to the shadow map or to the screen. At the same time, 4.7 million pixels are processed per frame.
Pixel load is kept low by excluding expensive post processing steps, and by not rendering particle effects.
Although the first graphics test is heavy on geometry, it features roughly 1/4 the number of vertices from GL/DXBenchmark 2.7’s T-Rex HD test. In terms of vertex/triangle count, even Egypt HD is more stressful than 3DMark’s first graphics test. That’s not necessarily a bad thing however, as most Android titles are no where near as stressful as what T-Rex and Egypt HD simulate.
Among Android smartphones, Qualcomm rules the roost here. The Adreno 320 based Nexus 4 and HTC One both do very well, approaching 60 fps in the first graphics test. The Mali 400MP4, used in the Galaxy Note 2 and without a lot of vertex processing power, brings up the rear – being outperformed by even NVIDIA’s Tegra 3. ARM’s Mali-T604 isn’t enough to pull ahead in this test either; the Nexus 10 remains squarely behind the top two Adreno 320 based devices.
Graphics Test 2
Graphics test 2 stresses the hardware’s ability to process lots of pixels. It tests the ability to read textures, do per pixel computations and write to render targets.
On average, 12.6 million pixels are processed per frame. The additional pixel processing compared to Graphics test 1 comes from including particles and post processing effects such as bloom, streaks and motion blur.
In each frame, an average 75,000 vertices are processed. This number is considerably lower than in Graphics test 1 because shadows are not drawn and the processed geometry has a lower number of polygons.
As you’d expect, shifting to a more pixel shader heavy workload shows the Galaxy Note 2 doing a lot better – effectively tying the Tegra 3 based HTC One X+ and outperforming the Nexus 7. The Mali-T604 continues to, at best, tie for third place here. Qualcomm’s Adreno 320 just seems to deliver better performance in 3DMark for Android.
The overall score pretty much follows the trends we saw earlier. Qualcomm’s Adreno 320 leads things (Nexus 4/HTC One), followed by ARM’s Mali-T604 (Nexus 10), Adreno 225 (SGS3), Adreno 305 (One SV), Tegra 3 (One X+/Nexus 7) and finally Mali 400MP4 (Note 2). The only real surprise here is just how much better Adreno 320 does compared to Mali-T604.
The purpose of the Physics test is to benchmark the hardware’s ability to do gameplay physics simulations on CPU. The GPU load is kept as low as possible to ensure that only the CPU’s capabilities are stressed.
The test has four simulated worlds. Each world has two soft bodies and two rigid bodies colliding with each other. One thread per available logical CPU core is used to run simulations. All physics are computed on the CPU with soft body vertex data updated to the GPU each frame. The background is drawn as a static image for the least possible GPU load.
The Physics test uses the Bullet Open Source Physics Library.
The physics results give us an indication of just how heavily threaded this benchmark is. The quad-core devices are able to outperform the dual-core Cortex A15 based Nexus 10, despite the latter having far better single threaded performance. The Droid DNA/Optimus G vs. Nexus 4 results continue to be a bit odd, perhaps due to the newer drivers included in the Nexus 4’s use of Android 4.2 vs. 4.1.2 for the other APQ8064 platforms.