Fitting a lens from straightness alone

Calibrating a lens usually means photographing a target: a printed chequerboard of known geometry, from a dozen angles, with the corners detected and the whole thing solved at once. It works well and it needs the target.

There is another route, and it needs nothing at all except a sentence about the scene: that edge was straight.

The idea #

A lens bends straight lines. So take the images of things that were straight in the world, undistort them with a candidate coefficient, and measure how straight the results are. At the right coefficient they are straight; at any other they are not. Minimising the straightness residual gives the lens.

This is called plumb-line calibration, after the vertical strings people used to photograph for the purpose, and it has three properties worth noticing.

It needs no measurements of the scene, only a topological fact about it. It needs no knowledge of the camera — not the focal length, not the position, not the orientation. And it separates cleanly from the rest of the calibration, which is why it can be run first on a picture whose intrinsics are unknown.

As a round trip #

The site’s foundation field is built on one habit: draw the picture from a known thing, forget the thing, recover it from the drawing alone, and compare. The camera recovered from the twelve edges it drew agrees with itself to one part in 10¹⁵, and the value of that number is not its size but the narrowness of the interface between the two halves.

This is the same shape on a harder problem. Straight image lines are distorted with a known coefficient. The coefficient is then set aside, and the fit is handed the bent point-lists and nothing else — no camera, no scene, no knowledge of what the coefficient was. It returns it to about 5e-15.

The interface here is narrower than the camera recovery’s, in fact. That one passes a list of line segments between the halves. This one passes a list of point-lists plus the assertion that each came from a straight edge, and the assertion carries no number at all.

The objective, and why it is bracketed #

The straightness residual is the root-mean-square, over the lines, of each line’s greatest departure from the chord joining its own endpoints. It has no knowledge of the scene in it whatsoever.

Minimising it is done here by nested golden-section search rather than by Gauss–Newton, and the choice is deliberate rather than lazy.

The objective is smooth and one-dimensional in each coefficient, and the brackets are known from the physics rather than guessed — a lens whose radial factor goes non-positive over the frame has folded the image over itself and is not a lens. A bracketed search cannot run away. And the two-coefficient objective has a long shallow valley, which is exactly the shape where a Newton step along the valley floor overshoots.

There is one mechanical detail worth recording because getting it wrong is subtle. The search steps into candidates for which the undistortion iteration does not converge — the folded-image case. For those, the objective returns infinity, and that is a legitimate objective value rather than a swallowed error: the candidate is saying “this lens has no straight lines at all”, which is infinitely bad. Returning zero instead would make the search prefer exactly the candidates the model cannot represent, and it would converge confidently to nonsense.

What the fit is actually looking at #

A useful way to see why this works at all: what changes, in the picture, as the candidate coefficient is swept?

At the true value every line is straight and the residual is zero. Move the candidate a little and the lines all bow the same way, because they are all being over- or under-corrected by the same radial factor. Move it a lot the other way and they bow the other way. So the residual as a function of the coefficient is a smooth well with a single minimum, and it has no local minima to fall into — which is why a bracketed one-dimensional search is not merely adequate but exactly right.

The depth of the well is set by how much the lines bend, which is set by how far from the principal point they pass. A line through the principal point contributes nothing at all, at any candidate coefficient, because a radial map cannot bend it. So a fit given only lines through the centre of the frame has a completely flat objective and no information whatsoever.

That is a real failure mode rather than a hypothetical one. Photographs of building interiors often have their strongest straight edges running through the middle of the frame — the horizon of a corridor, the line where two walls meet — and a fit given those returns whatever its bracket’s midpoint happens to be. The signal is in the lines that miss the centre, and the further they miss it the better.

The second coefficient, and the trouble it brings #

Everything above is the one-parameter fit, and it is clean. Free the second coefficient and something changes.

On clean data the two-parameter fit is exact: it returns $k_1$ and $k_2$ to the last bits. That is not the interesting part.

The interesting part is the shape of the residual surface around that minimum. It is a quadratic form, and its two eigenvalues say how sharply each combination of the coefficients is pinned. Here they differ by a factor of 813, and the correlation between the two parameters is −0.997.

What that means in practice: the data determines one particular mixture of $k_1$ and $k_2$ very well, and the mixture perpendicular to it very badly. Adding a substantial amount to $k_1$ and subtracting a compensating amount from $k_2$ leaves the lines almost as straight as they were.

So a fitted $k_1$ quoted on its own is not a measurement. It is one coordinate of a point in a valley, and moving along the valley changes it a great deal while changing the thing that was actually measured — the straightness — hardly at all.

What that means for a reported coefficient #

Three consequences, and they are the reason this essay exists rather than stopping after the round trip.

A coefficient without its conditioning is not reportable. Two calibrations of the same lens from different pictures will disagree about $k_1$ by far more than either’s residual suggests, and both will be right about the combination. Quoting the condition number alongside is the minimum honest report, and it is a number this machinery produces for free.

Fitting fewer parameters is often better. Pin $k_2 = 0$ and fit $k_1$ alone, and the answer is well determined — because the valley has been cut across rather than followed. The fit is biased if the lens really has a $k_2$, but a biased well-determined number can beat an unbiased ill-determined one, which is the standard bias-variance argument arriving from geometry.

And the wrong-model case is instructive. Fit $k_1$ alone against data from a lens that has both, and the fit returns −0.218 for a true −0.280 and leaves a residual of 0.68 px of sag. The residual is small — small enough to look like success — and the coefficient is 22% wrong. A residual that has stopped falling is not the same thing as a model that is right.

Where the ill-conditioning comes from #

It is worth seeing why the two coefficients are so nearly redundant, because the reason is geometric rather than numerical and no amount of better arithmetic removes it.

The radial factor is $1 + k_1r^2 + k_2r^4$, and the sensor covers a limited range of $r$ — typically from zero to about 0.6 in normalised units. Over that range, $r^2$ and $r^4$ are both monotone, both convex, and very nearly proportional to one another. Two nearly parallel basis functions span nearly a one-dimensional space, and the coefficients on them are nearly indeterminate.

Widen the range of $r$ the data covers and the conditioning improves, because the two functions separate: $r^4$ pulls away from $r^2$ at large $r$. That is the practical advice: use lines that reach the corners, not lines that stay in the comfortable middle of the frame. It is the same advice a numerical analyst would give about fitting polynomials on a short interval, arrived at from optics.

The slider on the figure varies the number of lines the fit is given, which is the other lever, and it is the less effective one. More lines reduce noise; they do not change the shapes of $r^2$ and $r^4$ over the range they cover.

The conditioning, differenced #

The condition number is measured rather than derived, and the way it is measured is worth a paragraph because it is a small instance of the site’s standing rule.

The objective runs through the undistortion iteration, which has no closed-form derivative worth writing. So the Hessian is differenced: three second differences and one mixed difference around the fitted point, assembled into a 2×2 symmetric matrix, and its eigenvalues taken analytically.

The check that the fit really is at a minimum comes for free from that: both eigenvalues have to be positive, and a fit that had converged to a saddle or a ridge would fail it. That is an assertion which has never fired here, which by this site’s habit means it needs a case where it would — and the case is the one the figure’s own refusal covers, where the model folds the image.

The residual is the report #

One habit worth carrying out of this essay, because it generalises past lenses.

The fit produces two things: a coefficient and a residual. The coefficient is what everybody quotes and the residual is what says whether the coefficient means anything.

A residual at the level of arithmetic noise — 4e-13 px here — says the model fits the data to the limit of the representation, which is the strongest statement available. A residual of 0.68 px, which is what the one-coefficient fit leaves on data from a two-coefficient lens, says the model is wrong rather than that the data is noisy. And a residual that is small but not tiny, on real data, says the model is right and the measurement was imperfect, which is the normal case.

Those three are genuinely different situations and they call for different responses — accept, add a term, report an error bar — and the only thing that distinguishes them is the residual. Which is exactly the argument the site’s recovery makes for reporting its bundle residuals rather than only its focal length, and the argument the refraction field makes for reporting the miss of its ray bundle rather than the fitted point.

Against the alternative #

It is worth being fair to the target-based calibration this essay is quietly arguing against, because it is better in every respect except the one that matters here.

A chequerboard gives metric information — the corners are a known distance apart — so it determines the focal length, the principal point, the distortion and the poses of all the views at once, jointly, which is what avoids the bias from solving them in sequence. It also gives many more constraints per picture than a plumb-line fit gets from a handful of edges.

What the plumb-line method has is that it can be run on a picture that already exists. An archive photograph, a frame from a film, a photograph of a building taken by somebody who is no longer available to hold up a target — all of them contain straight edges, and the coefficient falls out of them alone.

That is the same argument the camera recovery makes for reading a focal length out of a photograph whose metadata is gone, and the two together are a surprisingly complete toolkit for an uncalibrated picture: straightness gives the distortion, three vanishing points give the focal length and the principal point, and one known length in the scene gives the scale.

The name, and the practice it came from #

The method is called plumb-line calibration because that is literally what it was: Duane Brown, working in analytical photogrammetry, photographed an array of hanging plumb lines and fitted the coefficients that made them straight. A plumb line is the one object in the world whose straightness needs no measurement and no manufacturing tolerance — gravity supplies it.

The elegance of that is worth pausing on, because it is the same move this site makes repeatedly. The problem is that a calibration needs something known about the scene, and known things are expensive: a target has to be printed accurately, kept flat, and measured. A plumb line costs nothing and is straighter than any target, and the property it supplies is exactly the one the fit needs.

The modern version does not hang strings; it uses whatever the picture contains. Building edges, door frames, the join between wall and ceiling, the rail of a bridge — every one is straight to a tolerance far tighter than the distortion being measured. That is why the method survived the disappearance of the practice it was named for.

It also means the method has a bias that the string version did not: a chosen edge might not be as straight as assumed. The residual reports it, which is the point of the residual, and a fit whose residual will not go below a pixel is reporting that something in its input was not straight.

What it cannot do #

Two limits, stated rather than left for a reader to hit.

It cannot find the principal point. The distortion is radial about it, so in principle the centre of the radial pattern is determined by the data — and in practice it is very weakly determined, because moving the centre a little and adjusting the coefficients compensates almost exactly. That is the same valley, in two more dimensions. Serious calibrations solve for it and quote it with wide error bars; the site’s own three-vanishing-point recovery computes it exactly, from a different kind of information, and is the better instrument for it.

It cannot tell distortion from a curved subject. The method’s one input is the assertion that certain edges were straight, and if they were not, the fit will faithfully report the coefficient that makes them so. Photograph a barrel through a perfect lens, tell the fit its staves were straight, and it will hand back a barrel-distortion coefficient. That is not a defect; it is the method being exactly as good as its premise, which is the honest position for a calibration that carries no target.