Test, relevance and reputation

The November 2008 issue of Black & White Photography has a column in which Mike Johnston defends his way of testing cameras and lenses. He argues against trying to be completely objective: it is not the figures that matter, but the trial of the gear in dfficult, actual photography circumstances. I completely agree with Mike: too often testers focus on what is measurable rather than on what is relevant--not just in photography but also in my professional field of higher education. There, rankings are made of things that can be measured but whose relation to educational quality are uncertain. Similarly for photo gear: their 'performance' in laboratory circumstances is only a remote indication of what you can do with the thing in practice. In practice, all kinds of circumstances and especially your own needs are much more important than the 'objective' figures. You use different focal lengths, differnt distances, and handheld instead of on a tripod as they did in the test, or you take different types of pictures. For instance, my new standard zooom lens got fairly good test results, but with my preference for architectural pictures, the slight distortion in the wide-angle setting weighed more heavily than what I could read in the mags. Or take the high-quality lens I looked at: it was indeed great from a technical point of view, but way too heavy to be practical on a day-long hike. So let me quote Mike's conclusion: 'The bottom line is that observation is at least as important a method of inquiry as measurement is. It is no less relevant and no less reputable.'