The de facto industry standard to quantify lens performance is the modulation transfer function (MTF). An explanation of MTF is beyond the scope of this article, but excellent texts are available elsewhere [1, 2]. The MTF can be computed theoretically for known lens designs, but there are always implementation losses and the actual performance of a production lens falls short of the theoretical ideal. A more realistic performance indicator is measured MTF. For decades, the preferred measurement device has been the scanning-slit optical bench. However, such a system is not within reach of individuals and a new method has emerged with the advent of digital cameras. This method, which is popular among photo magazines and some enthusiasts on the internet, uses dedicated test charts and analysis software. Unfortunately there are disadvantages and pitfalls, which are rarely mentioned or even recognized, and lenses may be hailed or dismissed for the wrong reasons.
An optical bench is a calibrated measurement device designed specifically for the job. Measurements are automated, reproducible, independent of the operator, and allow a lens performance evaluation at infinity focus. This is not true for MTF measurements by means of photographing test charts (ISO, USAF, Siemens star, etc.) or at least to a lesser degree. Drawbacks of target lens tests include:
- Even with live view, focusing is less accurate than with an optical bench.
- MTF measurements at infinity focus are not practically feasible.
- The use of different cameras, targets, analysis software, as well as human factors render results irreproducible and difficult to interpret.
The last point is an important one. The measured MTF is not lens MTF, but system MTF, where the system is the entire chain of operator, lens, anti-alias filter or lack thereof, sensor response, in-camera processing, and post-processing algorithms. There are many factors that influence the final MTF values. The operator is part of the chain and influences the outcome, for instance via his ability to achieve a parallel set-up of target and sensor. Autofocus, manual focus, live view, … each method may yield different results. Some testers apply focus bracketing and simply use the image that yields the highest MTF value at a given position in the frame, an inelegant working method that is in glaring contrast with the calibrated optical bench and its standardized focus criterion. At the same time the system MTF can be seen in a positive light, because system MTF, including the operator, is what eventually determines the technical quality of a photographic image. An optical bench only provides the lens MTF.
One should be very careful with interpreting system MTF as an indicator of lens performance. Even a comparison between two lenses that uses the same camera, hardware and software settings, etc., should be judged with caution. For instance, differences between lenses may be reduced by digital sharpening algorithms or by the use of the luminance signal in the MTF analysis . On the other hand, differences between lenses may be exaggerated by testing at relatively close focus. Documentation of the test method and interpretation of the results are as important as the method itself. The following points can be used to assess the credibility of the test report:
- A lens has cylindrical symmetry, which implies that its rendering can be quite different for sagittal or tangential details. Test results are typically presented as MTF50 spatial frequencies, but only a single value is given per f-number and position in the field. If there there is no mention of the MTF50 being sagittal or tangential MTF (or some kind of average), the results leave a lot to the imagination of the reader.
- Sometimes frequencies are reported that exceed the Nyquist frequency of the sensor. Such results should be treated with great caution, especially if the sensor is equipped with an anti-alias filter — whose purpose is to attenuate frequencies above Nyquist. With dedicated targets and processing, it is possible to measure “something” beyond Nyquist, but such measurements are heavily influenced by the characteristics of the measurement device. The interpretation of MTF values above Nyquist as a property of the lens is uncertain, to say the least.
- Sometimes chromatic aberration is reported to increase with an increasing f-number. This is impossible. A smaller aperture reduces longitudinal chromatic aberration, whereas lateral chromatic aberration is independent of the aperture. What happens is that the lateral color fringing is mixed with the blur due to other aberrations. When the lens is stopped down, this blur is reduced and the lateral color becomes more clearly visible. The analysis software does not measure chromatic aberration, but apparent color fringing.
- Some testers state that field curvature is mitigated by specifically focusing on the corners of the target. This may increase the measured resolution, but cannot cure field curvature unless there is zero astigmatism. In the presence of astigmatism, the sagittal and tangential images are differently curved and cannot be in simultaneous focus. This brings back the first issue in this list. Which MTF is being optimized?
- For reproducibility and interpretation, the spectrum of the light source should be mentioned. For the same reasons, the employed camera and all relevant operating procedures, processing steps, and hardware and software settings should be given.
- The image magnification (or object distance) should be mentioned.
The first and the last point in this list are of particular importance. It is well known, to testers and readers alike, that lens performance is a function of the aperture. MTF measurements are therefore normally presented for several f-numbers. It is also well known that lens performance is a function of the position in the field. Borders and corners may be weaker than the image center. MTF measurements are therefore normally presented for several distances from the center, also known as image heights. It is less well known that there exists such a thing as sagittal and tangential lens performance, and that two MTF values are needed for each f-number and image height. Finally, it may come as a surprise that lens performance is a function of the object distance, especially for lenses with a large maximum aperture and/or an asymmetrical design.
The MTF parameter space is large. Even lens manufacturers present a limited amount of information in their product sheets. However, a minimum requirement to any MTF test report is sufficient documentation. For instance, there has to be mention of the object distance or image magnification. More often than not, this crucial parameter is missing in the presentation of results. As a rule, photo magazines do not bother to mention the subject distance. Internet sites may provide the information, but you need to dig deep. It is also possible that you encounter vague statements concerning the use of different chart sizes to best suit the lens, without the actually employed size being mentioned in the presentation of results.
All lens aberrations depend on the object distance. A simple yet convincing illustration shows how curvilinear distortion may change from moderate moustache distortion at long range, to strong barrel distortion at close range. Distortion does not directly affect MTF, but other aberrations do and they also vary with distance. (In addition, vignetting and the angle of view vary with the focus distance.)
Most photographic lens designs are optimized for peak performance at large distances. The reason is simply that this suits everyday use. For applications such as landscapes and cityscapes, it is desirable to have a flat field and good sharpness across the frame at a focus that is optically close to infinity. When an ‘infinity’ lens is used at closer focus, the optical performance suffers most in the corners. There are many close-focus applications where this is not very objectionable, for instance portraits or pictures of animals or flowers. The subject then is a three-dimensional object that is typically not placed in a corner.
User requirements are different for reproduction of flat objects like paintings, documents, and stamps. A requirement is a flat field with uniform sharpness across the frame, this time not at infinity but at close to intermediate object distances. Lenses optimized for infinity are not suitable for this task, but fortunately there exist dedicated lenses known as macro lenses. As expected, these perform less well at infinity. An elegant solution to combine the best of the two worlds is the use of floating elements. A lens design with floating elements has variable air spaces between two or more lens groups, which gives the designer more degrees of freedom for aberration control over the distance scale. Such a lens may perform well both at close focus and at infinity. Some lenses branded as a macro lens even have their peak performance at infinity, while still offering excellent image quality at close focus.
Since many lenses are optimized for large distances and lack variable air spaces, they are not suitable for reproduction photography. The question arises, then, why target MTF measurements treat these lenses as if they were reproduction lenses. The answer is simple. Testing at large distances requires an impractically large target and working space. Employed targets typically have a width between 1 and 1.5 m (information that is remarkably difficult to find). This corresponds to a magnification between 0.036 and 0.024 on a full-frame sensor, which is not the macro regime but which is not close to infinity either.
How big is the difference in MTF performance between image magnifications? The answer depends on the lens under consideration, but it can be substantial. As a case in point, Figs. 1 and 2 show MTF curves from a design study of two 2.8/25 retrofocus lenses that lack floating elements. These lenses were not taken into production and the MTF is theoretical, but that does not matter for the comparison. Furthermore the data are not MTF50 curves, but MTF as a function of distance from the image center for three spatial frequencies: 40, 20, and 10 line pairs per millimeter (lp/mm). The highest curves are for 10 lp/mm, the lowest for 40 lp/mm. The solid curves are sagittal MTF, and the dashed curves tangential MTF. For the interpretation of such graphs the reader is referred to . In the present study we adopt the simple viewpoint that higher is better. The objective is not to analyze and interpret each graph in detail, but to show how lens performance can differ between image magnifications.
Lens A in Fig. 1 is designed for a high performance at infinity focus, which corresponds to an image magnification m = 0. Graphs are also shown for m = 0.025, which corresponds to a subject distance of about 1 m and which is in agreement with typical magnifications of target MTF measurements. Comparison of curves shows that the sagittal MTF is not degraded much by reducing the focus distance from infinity to 1 m. In fact, at F/5.6 the rendering of sagittal details is more uniform across the frame. However, the overall lens performance is dragged down substantially by the tangential MTF. Starting at 10 mm from the image center, the curves drop rapidly and render corner performance much worse than at infinity focus.
In contrast to lens A, lens B is designed for a high performance at m = 0.025. Fig. 2 shows curves at F/5.6 for three values of the image magnification: m = 0, m = 0.025, and m = 0.050. Unsurprisingly, the best lens performance occurs at m = 0.025. Infinity focus noticeably lowers the performance for lens B at 20 and 40 lp/mm, and so does the close focus of m = 0.050. It is again the tangential MTF that suffers most from the refocusing. Candidate causes for a large separation between sagittal and tangential MTF are lateral chromatic aberration (the achilles heel of the retrofocus design) and astigmatism. The question arises again which MTF (sagittal, tangential, or an average value) is being published in target MTF reports. The inclusion of the graph for the full aperture of F/2.8 at m = 0.025 illustrates that the impact of focus distance can be as big as, or larger than, the impact of the aperture.
A low-cost lens MTF test has emerged by means of target reproduction photography and dedicated image analysis software. The method is valuable in that it yields measurements of system MTF. One can only admire the efforts that testers put in, and these efforts illustrate how difficult it is to get the most out of your equipment also in everyday photography. Unfortunately, the method also has some flaws. Since system MTF is measured and not lens MTF, the results are difficult to interpret. Moreover, the method yields lens ratings for reproduction photography, which for many lenses is not the intended application. The effect of object distance or orientation (sagittal or tangential) on MTF can be as big as, or bigger than, the effect of lens aperture or image height. MTF reports which include information on magnification and orientation are useful, so long as the limitations of the measurement method are kept in mind. MTF reports that lack the minimum required documentation are not very useful.
One also finds image quality assessments which do not derive MTF resolution values, but which simply show portions of the photographed target. These reports are more informative, but do not allow a quantitative comparison between different lenses. Still, the best way to find out whether a lens suits your needs is to use it for a while — if that is an option.