The lines that calibrate a lens

Fitting a lens from straightness alone recovered a lens’s distortion from nothing but the knowledge that some edges in a picture were straight, and then spent most of its length on why a coefficient recovered that way is not yet a measurement: the two radial coefficients are correlated at −0.997, and a fit given edges that all pass at similar distances from the centre is in a valley. Its advice came in two parts — reach the corners, for signal; spread the radii, for conditioning — and it said the signal lives in lines that miss the centre, “and the further they miss it the better”.

That advice was argued from the shape of the bend. This essay measures it, as a design question a photographer or a calibration procedure actually faces: given a few straight edges to use, where should they be? The answer has a part the earlier fit predicted and a part it did not, and the part it did not is the one that decides whether a calibration from an architectural photograph works.

Precision, stated as a number before any fit is run #

A fit on exact data recovers any coefficient exactly, which says nothing about how well it would do on marks placed by a hand or a corner detector. The question needs noise, and it has an answer that does not require running the noisy fit.

Mark each straight edge every 8 px along its visible length, and suppose each mark is placed with half a pixel of error in every direction. For each mark, the only part of that error that matters is the component across the bent curve — sliding a mark along its own edge changes nothing. Differentiate that across-the-curve position with respect to the lens coefficient, and with respect to the edge’s own position and angle, which the fit has to estimate too; assemble the derivatives into the information matrix; and invert it. The square root of its first diagonal entry is the smallest standard deviation of $k_{1}$ that any unbiased fit of that data can have.

The same matrix says more than one number. Its other diagonal entries are the precisions of everything else the fit estimates, and its off-diagonal entries say which of those estimates trade against each other — which is where the measured cost of each assumption found that a pixel costs different amounts depending on what is read through a fit. Here that trade turns out to be the whole story.

That is a prediction, and a prediction of a standard deviation is worth checking against one. Sixty noisy fits of one three-edge design, each a full plumb-line fit with every edge’s position and angle free, spread with a standard deviation of $5.2 \times 10^{-3}$ against a predicted $4.6 \times 10^{-3}$: a little wider than the bound, as a real estimator on sixty samples should be, and not narrower, which it could not be if the calculation were right.

One edge, and where it passes #

An edge through the centre carries no information at all about $k_{1}$. The information matrix is singular: a radial map moves every point along its own radius, and every point of a line through the centre is already on its own radius, so the line is not bent at any coefficient and nothing in its marks responds to one.

Move the edge off the centre and the precision rises steeply. At 5 px from the centre the best possible standard deviation of $k_{1}$ is $3.92 \times 10^{-2}$, comparable to the coefficient’s own size in a mild lens. At 180 px — near the top edge of this 400 px frame — it is $9.04 \times 10^{-4}$. On logarithmic axes the points fall on a line of slope −1.04.

A slope of −1 is what the bend predicts. An edge passing a distance $d$ from the centre is moved, at a mark a distance $x$ along it, by an amount proportional to $d(x^{2} + d^{2})$. The part in $d^{3}$ is the same all along the edge, so the edge’s own fitted position absorbs it; what is left, the bow, is proportional to $d\,x^{2}$. So the derivative of each mark with respect to $k_{1}$ grows as $d$, the information as $d^{2}$, and the standard deviation falls as $1/d$. Every horizontal edge in this frame spans the same 650 px of width wherever it passes, so the length of the edge plays no part in the comparison, and the small excess over −1 belongs to the terms that first-order argument drops.

So the earlier advice survives with a number attached. An edge twice as far from the centre is worth an edge measured with twice the precision, and four edges at the same small offset are worth about as much as one edge at twice the offset.

Three edges, when the centre is known #

A real calibration has several edges, and a design is a choice of where to put them. The figure compares four three-edge designs, drawn as small frames beside their bars:

• crowded on one side — three horizontal edges near the top of the frame, at 150, 165 and 180 px from the centre;
• across the centre — two of those near the top and one near the bottom, 180 px below;
• near the centre — three horizontal edges at 20, 40 and 60 px above it;
• placed by search — three edges at various angles, the best a seeded search of 300 random starts and coordinate descent found with the centre and $k_{2}$ free, rounded to the degree and the pixel.

With the distortion centre known, the ordering is the one the single-edge result predicts. The two designs with edges far from the centre are nearly tied — $5.83 \times 10^{-4}$ crowded on one side, $5.41 \times 10^{-4}$ across the centre — and the design near the centre is four and a half times worse, at $2.58 \times 10^{-3}$. Whether the three far edges are on one side or on both makes almost no difference, because with the centre known the only thing an edge contributes is its bend, and its bend depends on its distance from the centre, not on which side it is on.

If that were the whole story, “use edges far from the centre” would be complete advice. It is not the whole story, because a calibration from a photograph rarely knows the centre.

When the centre is free, one side is not enough #

The distortion is radial about a point, and that point is usually close to the middle of the frame and not at it. The principal point is not the centre measured what assuming otherwise costs a focal length, and a careful calibration frees the distortion centre along with the coefficient.

Free the centre, and the crowded design collapses: $5.69 \times 10^{-3}$, ten times worse than with the centre known. The across-the-centre design barely moves, from $5.41 \times 10^{-4}$ to $5.66 \times 10^{-4}$. The design near the centre is worse again, at $6.73 \times 10^{-3}$.

The reason is a confusion the edges cannot resolve on their own. A horizontal edge above the centre, bent by a barrel lens, bows upward at its middle. Move the distortion centre up toward that edge and the edge is nearer the centre, so the same coefficient bends it less; strengthen the coefficient and it bends more. Three edges all above the centre see the same trade from the same side, so a stronger lens with a higher centre fits them almost as well as a weaker lens with a lower one. The figure’s information matrix says how nearly: for the crowded design, $k_{1}$ and the centre’s height are correlated at 0.995.

An edge on the other side of the centre breaks the trade. Moving the centre up brings the top edges nearer to it and pushes the bottom edge further away, so the two bend in opposite senses, and no single change of coefficient compensates both. For the across-the-centre design the correlation between $k_{1}$ and the centre’s height drops to 0.29. Distance from the centre buys signal; straddling it buys the ability to tell the signal from a moved centre. The first was the earlier advice; the second is the part it did not have.

Where the centre itself goes #

The trade has a second victim that the scores for $k_{1}$ do not show: the centre’s own position, which a calibration usually wants for its own sake.

With the centre free, the crowded design leaves the centre’s height uncertain by 3.91 px, one standard deviation. The across-the-centre design pins it to 0.40 px, ten times better, and even the design near the centre does better than crowding, at 1.54 px. Free $k_{2}$ as well, and the crowded design’s uncertainty in the centre’s height grows to 28 px, while the across-the-centre design still holds it to 0.40 px.

Twenty-eight pixels is not a small number on this frame. It is larger than the 22.30 px that a sensor tilted 3° moves the principal point, which is a displacement that changes every conclusion a calibration draws about asymmetry. A calibration from three edges crowded near the top of a photograph cannot locate its distortion centre to within the size of the effects it would need the centre to diagnose.

When the second coefficient is free as well #

A full radial calibration frees $k_{2}$ too, and fitting a lens from straightness alone found that the two coefficients lie in a valley. The designs meet that valley in different ways.

With the centre and $k_{2}$ both free, the crowded design’s precision on $k_{1}$ is $6.93 \times 10^{-2}$ — a standard deviation a quarter the size of the coefficient itself, which is no calibration at all. The across-the-centre design gives $4.10 \times 10^{-3}$, and the design near the centre $1.65 \times 10^{-2}$.

Every design is much worse than with $k_{2}$ fixed, which is the valley: edges whose marks span similar radii ask $r^{2}$ and $r^{4}$ nearly the same question. The information matrix puts a number on each design’s valley. $k_{1}$ and $k_{2}$ are correlated at −0.997 crowded on one side, −0.990 across the centre, −0.978 for the searched design and −0.913 near the centre. Correlation is not precision — the design near the centre has the shallowest valley and still one of the worst scores, because its edges carry so little bend to begin with — but for designs with signal, the steeper the valley the more of it is spent.

And the ratio between the designs has grown. Crowded on one side is now seventeen times worse than across the centre, where with only the centre free it was ten times worse. Each additional unknown that trades against the coefficient is resolved by the same thing — edges on both sides, at a range of distances — and a design without that pays for every unknown at once.

The valley has a consequence beyond precision. A barrel model folds at a radius it sets itself walked along exactly this kind of valley and found coefficient pairs, indistinguishable in their straightness residual, that fold at 57.8° of field. A design that leaves the valley steep leaves a family of models that agree inside the frame and disagree about where the model stops being a lens.

How many marks an edge is worth #

Everything above spends marks every 8 px, which gives each full-width edge 82 marks and a three-edge design 246. Marks are cheap to add, and the information calculation says what they buy.

Halve the spacing to 4 px and the across-the-centre design has 489 marks and a precision, with the centre known, of $3.91 \times 10^{-4}$. Double it to 16 px and it has 123 marks and $7.36 \times 10^{-4}$. Each halving of the spacing improves the precision by close to $\sqrt{2}$ — the law for independent errors, where information adds mark by mark.

That law is exactly what the noise model assumed, and it is where the calculation stops being a guide. A floor with a referent measured more points along each line for a focal length recovered through a lens, and found the error falling, turning, and rising to a floor that was the model rather than the noise. The bias out of reach found a spread falling as one over the root of the number of readings while a bias did not fall at all. Denser marks on an edge buy down the spread of $k_{1}$ and nothing else. If the edge is not quite straight in the world, or the marks are all displaced the same way by the edge’s own blur, every extra mark confirms the same wrong answer more precisely.

The practical reading is that marks are the last thing to add. A design that straddles the centre at a range of distances buys a factor of ten over one that does not; halving the mark spacing buys a factor of 1.4, and only against the noise.

The searched design, and what a search does not prove #

The fourth design was found by searching, with exactly this freed set, for the three edges that minimise the standard deviation of $k_{1}$: 300 random starting designs and a coordinate descent from the best, stopping when no step of a tenth of a degree or a quarter pixel improved it. It gives $4.60 \times 10^{-3}$.

The obvious design — two edges at the top, one at the bottom — gives $4.10 \times 10^{-3}$, eleven per cent better than the result of the search for the best.

That deserves stating rather than smoothing over, because the temptation is to report the searched design as optimal. The search found a local optimum of a function over six variables with a long valley in it, from a finite set of starting points, and rounded the result; the obvious design sits in a different basin that the search did not enter. A searched design is an upper bound on how badly the best design does, not the best design. In this case the upper bound was not tight, and a hand design guided by the argument above did better than the machine guided by nothing.

Why the best edges are the ones a photograph rarely has #

It is worth turning the result toward the kind of picture a plumb-line calibration is most often run on: an architectural photograph, where straight edges are free.

The strongest straight edges in a photograph of a building’s interior are the horizon of a corridor, the line where the ceiling meets a wall, and the vertical of a corner — and all three tend to run through or near the middle of the frame, because photographers compose around them. Those are the edges that carry nothing. The edges that carry information are the ones near the frame’s boundary, and in a composed photograph they are often only on one side: the top of a façade against the sky, with the ground cropped; a row of window heads, with nothing matching them below.

So the design this essay finds best is exactly the one a photographer is least likely to supply by accident, and the design it finds worst — edges crowded on one side — is the one a cropped architectural photograph most often does. The practical reading is not that such photographs cannot be used. It is that a calibration from one should free the distortion centre only if the edges straddle it, and should otherwise hold the centre and report that it did, because which reference is measured from decides the answer more than how carefully it is marked.

What these numbers do not settle #

The noise model is isotropic and independent. Each mark is placed with half a pixel of error in every direction, independently of its neighbours. A corner detector’s errors are often correlated along an edge, and an edge that is not quite straight in the world is not noise at all: it is a bad row, and a wrong match is not a small error found that least squares has nowhere to put a bad row except across all the others. The predicted precisions are the best case for the stated noise, and the sixty-trial check shows a real fit reaching them, not beating them.

The frame and the lens are one arrangement. A 78° frame, a coefficient of −0.28, marks every 8 px. The slope of −1 and the collapse of one-sided designs when the centre is free are geometric and should transfer; the individual standard deviations are specific to the arrangement. A fisheye is a different family again, and which rule a fisheye obeys from straightness alone is where straight edges are asked to choose a law rather than a coefficient.

And only $k_{1}$ and the centre are scored. A design could be good for both and poor for $k_{2}$. The across-the-centre design is good for everything measured here, but a calibration that cares about the second coefficient — which decides where the model folds — should score it directly rather than infer it.

Still open: which marks along an edge carry the information #

Whole edges were placed here; the finer question places marks along them. The information an edge carries about $k_{1}$ is not spread evenly along its length. The bow grows as the square of the distance along the edge from its closest approach to the centre, so a mark near that closest approach responds little to the coefficient, and a mark toward the ends responds much more. The measurement that answers it computes the information contributed by each mark along a single edge, finds what fraction of an edge’s worth lies in its outermost quarter at each end, and measures how precisely $k_{1}$ is determined when marks are spent only there instead of evenly — which decides whether a calibration from a photograph should chase the full length of every edge or only its two ends, near the frame’s corners where a lens’s marks are hardest to place.