New way to compare mobile phone cameras: Benchmarking takes into account also camera speed

According to the doctoral dissertation, heavy image quality algorithms used nowadays may slow down functionality of mobile phone cameras. Moreover, the research highlights the main principles to compare mobile phone cameras. For example, cameras should be measured in different light environments. defines a new benchmarking method for mobile phone cameras. The benchmarking combines different image quality and camera performance metrics to a comparable, single value benchmarking score. According to the research, heavy image quality algorithms used nowadays may slow down functionality of mobile phone cameras. Moreover, the research highlights the main principles to compare mobile phone cameras. For example, cameras should be measured in different light environments.

Veli-Tapani Peltoketo's doctoral thesis defines a new benchmarking method for mobile phone cameras. The benchmarking combines different image quality and camera performance metrics to a comparable, single value benchmarking score. According to the research, heavy image quality algorithms used nowadays may slow down functionality of mobile phone cameras. Moreover, the research highlights the main principles to compare mobile phone cameras. For example, cameras should be measured in different light environments.

According to the latest statistics, over three billion digital images are shared every day in different social media platforms. A significant number of the images are captured using mobile phone cameras. Even though the image quality of mobile phone cameras is clearly improved, there are differences between cameras. Also the performance i.e. speed of camera functionality is varying between cameras.

When the number of camera features is increasing, comparison between cameras is more and more difficult. Should a consumer select a camera including the best color fidelity or noise characteristics? Or is pixel amount or video quality the right selection criteria? Quite often one superior feature of a camera reduces quality of other features which makes the comparison even more difficult.

"Image quality is not the only entity which should be taken into account when mobile phone cameras are compared. Sophisticated image quality algorithms combined to huge pixel amount slow down the functionality of mobile phone cameras," says Veli-Tapani Peltoketo, who defends his dissertation at the University of Vaasa.

Nowadays an image captured by an imaging sensor is practically rebuilt again in a post processing pipeline of a digital camera system. The doctoral thesis reveals cameras which image quality features are superior but correspondingly the same camera can be one of the slowest of the comparison.

"Even though the image quality of a camera can be extremely good, slowness of the camera may prevent to capture the right moment and the result is as same as an image with poor quality: the image is not used," says Peltoketo.

Breathtaking speed of camera development challenges measurement methods

Nowadays, the image quality is one of the top research area of digital imaging. Several new image quality standards have been published and old standards are updated continuously. A breathtaking speed of camera development requires new methods and metrics to measure quality of new features. However, there are no standardized benchmarking metrics which include all, or at least the most important features and offer a comprehensive and simple benchmarking result.

To build up a generic evaluation of a camera, it has to be decided which parameters are used and compared. The doctoral thesis includes a comprehensive research for image quality parameters, metrics, and artefacts of mobile phone cameras. Moreover, the thesis contains several speed metrics for mobile phone cameras.

If different metrics can be combined to a single value score, comparison between cameras is easier. However, if all image quality and speed metrics are listed, there are more than fifty different metrics. Combining such a number of metrics to a single score is not reasonable. The thesis investigated different metrics and ended up to use spatial resolution, texture resolution, color fidelity, noise, and speed metrics to create a novel benchmarking metric. Using these metrics a generic and comparable benchmarking score is calculated.

Publicity of measurement methods is essential

The research introduces also main principles which should be taken into account when benchmarking scores are implemented for mobile phone cameras.

Firstly, all measurement methods and equations have to be public. Used metrics, measurement methods, and especially equations may include intentional or other weights which can bias the result.

Secondly, different measurement environment should be used when mobile phone cameras are benchmarked. Especially low light environment is very challenging to mobile phone cameras. Image quality and speed results may vary significantly in low light environment.

Finally, a perceptual image quality should be taken in to account when benchmarking is done. A color saturation is a good example of differences between perceptual and objective metric. An image looks better, if the color saturation is artificially increased. However, objectively the image includes a color error.

One may ask which is the best mobile phone camera? Peltoketo notes that it is quite difficult to give an unambiguous answer. However, if the results of the thesis give some answers of cameras available year 2014.

"The best image quality was measured from Lumia 1020 and Zopo C2 cameras whereas the speed performance was best in iPhone 5s Samsung S3 cameras."