OUR PROCESS

INSIDE OUR TESTING METHODS

INTRO

BY: MARK LaFLEUR
Our goal was to put together the most comprehensive, controlled, unbiased and useful anamorphic lens comparison test possible. It’s impossible for many DPs who don’t have the time or access to these lenses to be able to do a thorough side-by-side comparison in order to choose the right lens for a job. Some of these lenses are either so expensive, or so rare, there isn’t really a rental house you can go to where you could test them all under one roof. We gathered our lenses from multiple rental houses as well as several owner/operators.
In anamorphic lenses there are many options, and since these lenses inherently have so much character, it’s really important to test before you choose a set of lenses for a given project. We wanted to create a resource that was so thoroughly thought out and executed that a DP would confidently be able to choose the right lens using this test alone.
Thanks to the Internet, we live in an age where there is a wealth of information that is so easy to access. However there is as much misinformation out there as there is good, useful information. Lens test footage exists, including some that are very comprehensive, but even the good tests don’t always have enough lenses or focal lengths. Most don’t have all the testing we would want to see or only tested each lens at 1 T-stop. There are other factors to consider too. Were the lenses they used recently serviced? Did the shooter actually nail their focus or did they miss it and that’s why the image looks soft? Does the video look bad because of low output resolution or heavy compression? Did they color correct, sharpen or manipulate the footage in some way?
Our goal was to put together as many sets of lenses as possible, test them in as many useful ways as possible and present as much information as we could so that DPs can feel like they are making an informed decision. In some ways we feel like this test is even more useful than going to a rental house and testing the lenses for yourself. In that setting, you are stuck with whatever lighting is in the rental house and you can’t necessarily put all the elements in your frame that will show you as much of a lens’ character as possible. Also, it might be hard to convince a rental house to let you test lenses for two and a half days straight or convince a friend to be your model for a 3-day weekend! Between our thorough approach to testing and our Quad Video Player, which lets you simultaneously compare 4 lenses at your choice of T-stop (or the same lens at 4 different T-stops!), we think there is no better way of selecting the right anamorphic lens for your project.
We tested 13 sets of anamorphic lenses as old as the late 1950s and as new as a soon-to-be-released prototype lens. We tested the most exotic, high performance, expensive lenses available, right along side an inexpensive anamorphic adapter that screws on the front of an old still photo lens. We focused mainly on 2x front anamorphic prime lenses, but also included one 2x rear anamorphic zoom, one front anamorphic 1.5x zoom and as mentioned, a 1.5x front anamorphic adapter.
We wanted to show all aspects of the character and performance of the lenses and we wanted to show how that can change sometimes quite drastically depending on what T-stop you are shooting at, so most lenses were tested at 3 T-stops, faster lenses were tested at 4 and slower lenses were tested at 2 or 1 for our slowest lens. We did not test any lens at apertures smaller than T4. There were two big reasons for this. Each lens had to go through our “real word test,” our focus chart, our distortion chart and our projection room. With about 40 individual lenses at multiple T stops, we had to conduct nearly 400 rounds of individual tests. If we had the time and budget we would have tested them at T5.6 and T8 too, but due to time and budget, we had to draw the line at T4.
We shot our tests on ARRI Alexa Mini cameras. We chose the Alexa Mini because ARRI is still regarded as the industry standard for digital capture and because the Mini has a 4:3 sensor which is the best way to take advantage of 2x anamorphic lenses.
We recorded in Open Gate, Log-C and used Apple ProRes 422 HQ codec for compression. Since we knew these images were going to live on the internet and therefore be compressed, we did not see the point in recording ARRIRAW or ProRes 4444, which would have added unnecessary costs to our post production budget.
We are presenting the images in 2:40:1 aspect ratio with ARRI’s Rec709 LUT applied to the image. No other color grading or sharpening has been applied to the images. We adjusted exposure using the camera’s shutter. We could have adjusted exposure with our lighting, but we did not have the resources to use lights powerful enough to scrim them down 3 stops, and that would have taken forever to adjust every light on set each time we change our T-stop. We could have used ND filters in theory, but that introduces more glass to our image, which we didn’t want. Adjusting the camera’s shutter was efficient and in our opinion affected the image the least. Also, since we had a lot of subtle variance in maximum apertures (T1.3, T1.4, T1.6, T1.9, T2, T2.1, T2.2, T2.3. T2.4, T2.5, T2.8, etc.), being able to accurately dial in the exact mathematical shutter angle allowed us to get perfect exposures, which is so necessary for a test like this.

THE "REAL WORLD" TEST

Our main test set-up is what we feel is more of a “real world” shooting situation.  It’s difficult to create one set-up that has every element or challenge a DP encounters, but we wanted to include as many as we could within one set-up, that is a completely controlled environment. Nothing in the frame is there randomly. Each element including the building and where we placed the model in relation to it was done with purpose and to show as much of a lens’ character as possible.
The Location
We chose a location with enough depth so that we could see where the walls meet the floor and the vertical lines of doorways, windows, and corners of the room. All these straight lines really help show how much distortion a lens has in a “real world” shooting situation. We placed our model in between two windows, which incorporate an architectural design with many straight vertical and horizontal lines so that we can still gauge distortion on the longer focal lengths where we can’t see the straight lines where the walls meet and where the back wall meets the floor.
The Subject
First and most important, we have a person in the shot. Since one of the most common things we place in front of a camera are faces, our main test had to have a person front and center. We lit her in a fairly classic way with a soft key light coming from camera right, slightly above her head at a 45-degree angle. She has a back-light opposite the key-light and a soft fill coming from camera-left, also from a 45 degree angle. All lighting on her is incandescent ARRI fresnels, through Chimera soft boxes. No lights were dimmed down at all to keep the color temperature neutral. The model is wearing a white shirt that has texture. The reason for the white shirt was to help show the color cast of a lens. Also, the fine detail in the shirt shows a lens’ sharpness at the edge of the frame. You’ll notice with some lenses the shirt has no texture at all and on others you can really see the texture and design.
The placement of the camera in relation to the model was very important to us. You will find some tests where the camera’s distance doesn’t change from the model. So on wider lenses you might see a waste up shot where the model’s face will be small in the frame. By the time you get to the longer focal lengths, the subject’s face is much larger in the frame. In our minds there are no rules about what focal lengths are allowed to shoot someone in a close-up. Wide-angle lenses should not be limited to head-to-toe shots. Also, in a test it’s nice to have your subject relatively close to the lens to more accurately gauge a lenses sharpness and character. Some lenses are so sharp they can make skin imperfections evident in even the most perfect of complexions. We chose to place the camera relative to the subject so that the model was always in a traditional head-and-shoulders close-up. Not only can you see just how much detail there is in the model’s eyelashes, skin and hair, but you can also see how much a lens distorts her face. All lenses especially wide-angle and telephoto lenses stretch or compress and often distort a person’s face. Sometimes in a flattering way, other times in a very non-flattering way. Anamorphic lenses are known for the way they sometimes distort faces in close-ups. The term “mumps” was coined to label this effect. I think this is something that is really important to pay attention to in these tests. Some lenses stretch a person’s face the closer it is to the camera, while others make faces thinner. This is an important characteristic for every DP to consider.  No two lenses in this test distort our model’s face the same way.
Turning the Model's Face
We had the model turn to the left and to the right. This was to show how much a lens distorts her facial features in a 45 degree angle as well as in profile.  Since we often don’t shoot people straight on, these are actually more practical real world angles to shoot your subject than looking directly into camera.
Practical and Props
We included a continuous line of incandescent rope lights so you can see the bokeh of pinpoint lights sources and how it changes depending on where it is in the frame from edge to edge. You can see how the bokeh morphs and changes as we pan and tilt as well as any chromatic aberrations that might exist when we focus rack through the lights.
We included a few practical light sources for a variety of reasons. There are two bare bulbs on frame left. One is clear glass so you can see the filament and the other is frosted white. These show us if a lens is prone to bloom. They can also show us chromatic aberration on the focus racks, and they did cause lens flares on some lenses, which is a great indicator of just how susceptible a lens is to flaring. There are 4 china balls hanging from the ceiling. Again these show us if a lens has a tendency to bloom, but they also show us the way distortion effects round imagery. This may seem like a small detail, but this stuff is really important. For instance if you are shooting entirely in the woods, a lens’ distortion will not be that distracting since the organic shapes of trees and plants don’t follow patterns and rules like straight lines or geometric shapes. Since our china balls were almost perfectly round, we can see how some lenses bend them into ovals or shapes that look more like eyes. 
Lastly we had the two orange lamps on frame right. Each lamp has a different kind of texture and pattern and both lampshades have tightly textured surfaces.  We placed these closer to the edge of frame. Just like the texture in the model’s shirt, some lenses will reveal the texture of these props. As we focus rack past the lamps pay attention to the lamps and see if you can see the textured surface of the lamps shades. The tall, skinny lamp also gives us another straight line which helps show how much lens distortion we will see on the edge of frame.
We included a microphone in the lower left corner of the frame. This prop shows us a few things. The microphone is placed at minimum focus so that we can see how close each lens can focus. Since a microphone is a fairly familiar object we have an idea of its scale, so it gives us a good sense of what lenses would work well for close-up work without the use of diopters. Testing close focus also shows just how much we can throw the background out-of-focus. We chose a microphone that had a very shiny, almost reflective surface butting right up against the cutouts, which are almost black. This is another way to show if a lens is prone to chromatic aberration. And finally since the microphone is close to the bottom left corner of frame, it shows you how sharp a lens is out to the extreme corners at close-focus.
Focus Racking
We focus each lens to infinity and then to minimum focus. The back wall is not at infinity, so it’s important to gauge sharpness as the lens racks through the wall, not when it rests on infinity at which point nothing in the frame is in focus. We did this so we can see how much we can throw foreground objects out of focus and what foreground bokeh looks like at the extreme. We then rack to minimum focus to show a lens’ close focus and to show what the background bokeh looks like all the way to its most extreme. It’s important to watch different areas as we rack through them. We can really see how sharp a lens is as it racks through the model, the lamps and the wall. We can really see chromatic aberrations on the practical lights and high contrast areas of the frame as focus racks through them. And of course, and maybe most important to some DPs, we can see the way a lens breaths when focusing.
In spherical lenses, breathing is pretty straight-forward. Lenses breath or they don’t and the effect looks like the lens is zooming in and out. With anamorphic lenses, breathing is not so simple. In the ARRI / Zeiss Master Anamorphics, the lenses don’t really breath at all. The frame stays the same, and the geometry of everything in frame stays the same. Nothing is squeezed or stretched. The lenses behave like ARRI / Zeiss Master Primes. In some vintage lenses like the Kowas, there is a zooming effect as well as a stretching or squeezing of the image and it’s not quite equal on the vertical and horizontal axis. Panavision lenses do not breath on the horizontal axis. There is breathing on the vertical axis only, something that is a patented trademark of all of their front anamorphic lenses. Lenses like Lomos, certain focal lengths of the Todd AOs and the Cineovision breath more on the horizontal access and it changes the geometry of the room making it look like the walls are bending. It’s lenses like these that really deserve the term “breathing” since it’s not a zooming effect as much as it’s a distorting effect that makes it look like the walls are moving. It really feels like the frame is breathing in and out. It’s an effect that can be used to your advantage and like Panavision anamorphic lenses, it’s a signature look that spherical lenses can’t duplicate. Since most of these lens sets each utilize different focus systems (some very similar others completely different), they all tend to breath differently.
Panning and Tilting
There are many effects to look out for when we pan and tilt the camera. It makes the distortion of a lens really obvious especially in geometric shapes like doorways, windows, where walls meet and where the back wall meets the ground. One very obvious characteristic is how focus falls off from center to the edges of frame. We can see in many of the lenses the model’s face gets soft and often stretched or squeezed at the edges of the frame. This is important when choosing the right lens for a project. If you plan to have the subject near the center of the image, a vintage lens might be a good choice since the natural softening of the edges and barrel distortion will make a viewer focus even more on the center of the frame.  On the other hand if you are shooting a project and you will have shots with 2, 3 or 4 actors appearing shoulder to shoulder in frame, you may need a lens with more consistent sharpness from side to side in order to keep everyone sharp.
It’s important to look at the bokeh as we pan and tilt since bokeh on most of these lenses looks very different form the center of frame vs. the edges. A lens’ tendency to vignette is more obvious in a panning or tilting shot vs. a static shot. Watch as the exposure changes on the model’s face as she travels from center over to the edges. In some lenses the exposure difference is quite dramatic.
Lens Flares
One of the biggest draws to anamorphic lenses seems to be the flares they can produce. We turned off most of the lights in the room so that the flares could stand out as much as possible. We used an LED flashlight on its low setting about 10 feet from the camera. We made sure to hit the lenses from all four corners and in the center of the lens since the lenses react differently depending on the angle of the light source and where it is in the frame. We added a second flare test that includes an iris rack since the lens flares change dramatically based on your T-stop. We find the iris rack really important since many of the lenses especially the vintage lenses have severe veiling flares and other artifacts that completely take over the frame when shooting wide-open, but become much more subtle when stopped down.
Color Chart
We shot a color chart for those who want to really scrutinize how well the lenses match from focal length to focal length in a set or to compare them set to set. The chart was placed in the same lighting as our model and placed on her mark so the intensity of the light is consistent.

FOCUS CHART

Shooting focus charts shows us a lot of useful information. Our charts were provided by Duclos Lenses and they were designed specifically for anamorphic lenses. We asked Duclos to print them big enough to be mounted to 8’ x 4’ pieces of wood. The reason for this was because we were testing some very wide-angle lenses, and if the charts were too small we would not be able to fill the frame with the chart due to the limited close focus capabilities of some anamorphic lenses. It worked well, since even on the widest lenses we were able to keep the camera about 6 feet from the chart.  One disclaimer. Since we had to glue the charts to huge pieces of wood, the glue did not dry perfectly on the sides. This resulted in a couple of wrinkles at the edges of the chart. If you notice those little wrinkles, please note it is not an issue with the lenses. This was human error that occurred when we mounted Duclos Lenses awesome charts to our wood structures. Most of the surface area of the chart was perfectly flat. We apologize for this mistake.
The fine detail of the focus “targets” really help illustrate the sharpness of a lens from the center all the way out to the corners. The chart shows us a lot more than sharpness though. Because of the high contrast of black ink on white paper this test also shows us if a lens is particularly prone to chromatic aberration. Since we lit the chart as evenly as possible, it’s easier to see how much each lens vignettes. We focused the lenses to infinity and to minimum focus, which should be another good indicator of a lens’ breathing characteristics. Lastly, since the chart is printed on brand new, perfectly white paper, you can get a good idea of the color cast of a lens. We shot most lenses at 3 T-stops. The faster lenses were shot at 4 T-stops and some of the slower lenses were shot at only 1 or 2. It’s going to be very interesting to compare the same lens at multiple T-stops on our “quad player” which will really highlight how much a lens changes as you stop down.

DISTORTION CHART

Duclos Lenses also provided us with our checkerboard distortion chart. This simple pattern is the easiest way to show very clearly a lens’ distortion characteristics. Most front anamorphic lenses produce barrel distortion, but pay close attention to this test because a few lenses caused some unique pincushion distortion and others have very complex distortion where it’s actually a combination of barrel and pincushion that changes from the center of the lens out to the edges.  We shot all the lenses at T4 only since the purpose of this test was to show distortion only.

PROJECTION ROOM

Our final testing area was the projection room. Lens technicians and most rental houses use projectors as the main way to assess a lens’ performance. Rather than light passing through the front of a lens onto film or a digital sensor, a lens projector shines light through a lens from the rear element out the front element onto a flat white surface. So you are seeing exactly what happens to a beam of light as it passes through the elements of a given lens. Precision cut patterns are placed between the projector bulb and the lens so that a test pattern is projected through the lens. These patterns are finely detailed and they can really show you a lot about a lens. Even better than our focus chart, a lens projector can show you the sharpness of a lens. In fact, there is no better or more accurate way to visually assess a lenses sharpness than a projector.  You can easily see chromatic aberration more accurately than any real world test because you are seeing it in its purest form essentially with no imagery or color to distract you. Since you can focus the lens while it’s on the projector you can easily examine focus breathing.

We were excited to use the projector to find out what T-stop each lens is at its maximum performance. You can also see if a lens is softer in one corner, if there is image shift, if an element is out of alignment, the size of a lens’ image circle. Basically you can see a lot. The problem is you really need to be in the room to appreciate the most of the data.
For our test we took photos of the projected image. That sounds useful, but there are some issues and limitations to it. It’s like seeing a picture of the Grand Canyon as opposed to actually being there. Since taking a photograph means you are recording the lens’ projected image with another lens, you are introducing the characteristics of that lens onto the projected images. So the photo we take is going to be affected by the sharpness, distortion and character of the lens we used for the photos. That means the data is immediately skewed.  Also, even if we somehow had a “perfect lens” that was infinitely sharp and had zero distortion and perfect illumination from center to corner, we would need a camera with a sensor with a HUGE pixel count to be able to record fine enough detail to show how many line pairs a lens can resolve. In theory we could have taken macro shots of several areas of the projected image, but we were at the mercy of time limitations, and we had so many lenses to test. So we opted to sum up our findings by the experts that were in that room. We came up with a simple system of rating each lens. The data is presented by Jorge Diaz-Amador, an expert lens technician who was in charge of the projection room testing.
A FEW WORDS BY: JORGE DIAZ-AMADOR
"An initial test was done on the Lumacon to determine the T stop for best sharpness/contrast in the center of the image. Projection tests were conducted with a P+S Technik / Kish Star LED illuminated lens test projector using the Otto Nemenz reticle. Each lens was tested once at maximum aperture and once at optimum center contrast aperture. The projected test pattern was observed and audio notes taken. A Sony a6000 camera was custom mounted directly above an on-axis with the P+S test projector to provide a reference image."

CONCLUSION

No test is perfect. If we had the time and resources we would have done even more tests in all sorts of real world situations. But we hope this test gives enough fair, accurate, unbiased data so that filmmakers will be able to make an informed decision about what lens they decide to use for a given project.
Test All Lenses