There's more to smartphone camera testing than test cards and statues

I may get mocked for my 'party' mock-ups when testing smartphone cameras, but my tests represent a better look at real world photos, i.e. of people indoors. Moreover, I also take into account facilities like lossless zoom, whereas this slightly questionnable set of test results from the usually reliable DxO mark folks shows the new Apple iPhone 6 models to both be top of the tree, with the classic Nokia 808 in 6th place and the newer Nokia Lumia 1020 down in 10th place overall! Remind me to take the DxO testers down the pub sometime and explain how to really test phone cameras....

Now, don't get me wrong, there's benefit in looking at colour balance and accuracy in photos, and the DxO folk do a good job at this. And yes, the Lumia 1020 (and Nokia 808 before it) can sometimes get these factors wrong. But here's DxO's latest overall chart:

DxO's chart

Now look, there are other factors used in real life which just aren't taken into account by DxO's scoring system:


I'm no fanboy, mind you. I'm not drinking the Nokia/Microsoft kool-aid (well, maybe a tiny bit - it's tasty). The 1020's shot to shot time is too slow and the camera application is too slow to start (note that the older 808 was a lot faster, thanks to a dedicated imaging signal processor). But, if we're concentrating on absolute image quality, as the DxO folk are, then camera speed shouldn't matter. And it beggars belief that the Xenon flash and zoom functions on the Nokia 808 and Lumia 1020 weren't considered at all in the scoring.

It should also be noted that the DxO system also factors in video capture performance, an area where I'd agree that Nokia doesn't automatically top the field, though the PureView zoom works even better for video than it does for stills. And, despite the 1020 having OIS, the Snapdragon S4 chipset isn't up to capturing 4K video or handling super-fast frame rates for special effects, so it's not surprising that the (808 and) 1020 fall slightly  behind here. But the OIS does produce very decent video capture, the 1020's no sluggard in this regard, plus on both the 808 and 1020 you get the stereo Rich Recording HAAC mikes that can cope with the loudest rock gigs.

Lumia 1020 and 808 PureView

So is there a really reliable smartphone camera test that includes flash performance on moving subjects? And maximum detail when zoomed? Nope, at least not to my knowledge. I do try in my own camera head to heads to include these factors among the test scenes, but I suspect I'm in the minority - most reviewers shoot testcards that don't laugh and ignore the zoom aspects of Nokia's PureView technology - go figure, on both counts.

For my part, I'll be putting the Xperia Z3 and Z3 Compact head to head with the Nokia 808 and Lumia 1020 in the next week, watch this space. And, I promise, no testcards or statues. You might get a steam train or two and a laughing guy with a beer bottle though!!

Published by Steve Litchfield at 6:39 UTC, September 26th

Section: Articles
Categories: Comment, Hardware
Platforms: General, Symbian^3, General, Windows Phone 8