A square closes the line of cameras

A box leaves a line of cameras measured what a picture of a box holds when pixels need not be square: a one-parameter family of exact cameras, each with a different width, focal length and principal point, whose right angles all come out correct. The centre closes it, at the cost of having the centre. But the point of reading a squeeze from a picture is usually that the camera has gone missing, in which case its centre has gone with it, so what is wanted is an extra fact that comes from the scene.

The pixel that is not square had already observed the answer in the abstract: a viewer who sees a squeezed face is using a known shape as a fourth constraint. A face is a difficult instrument, but the same purchase can be made with simpler things. The simplest is a square.

A fourth right angle #

Every condition this recovery uses has one form. Two directions at right angles in the world vanish at two points $p$ and $q$ on the page, and those points satisfy

$$\frac{(p_x - c_x)(q_x - c_x)}{a^2} + (p_y - c_y)(q_y - c_y) + f^2 = 0.$$

That is a single equation in $a$, $c_x$, $c_y$ and $f$, and it is linear in a suitable rewriting, $1/a^2$, $c_x/a^2$, $c_y$ and one more. Three pairs from a box supply three of them, one short. So any further pair of directions known to be at right angles, and not already implied by those, closes the system.

Put a square on one of the box’s faces. Its diagonals are at 45° to the face’s two edge directions and at 90° to each other, so they are a fourth perpendicular pair. Are they really a new condition? It is natural to doubt it, since the diagonals’ two vanishing points lie on the line through the face’s two edge-direction points: a square is only a pair of points on a line that is already there. But the right-angle condition is a pairing on that line, whose fixed points are complex. One pair determines one equation about that pairing and a different pair determines a different one. It takes two pairs to fix the pairing. A rectangle’s edges supply one and the square’s diagonals the other.

From exact marks, all four numbers come back exact: width 1.330, focal length 854.0, centre (345.0, 210.0), each to about ten digits, in a camera the solver was told nothing about. The linear form means no iteration and no starting point. There is a refusal worth recording here too: hand the solver a pair already implied by the first three, the same right angle listed twice, and its system has no unique solution, and it says so. The closure comes from the new equation and nothing else.

A circle does a closely related thing, in place of the square. The picture of a circle on the face meets the face’s horizon at the plane’s complex circular points, and one conic calibrates the camera showed what those are worth. They are the pairing’s fixed points themselves. So a circle gives the pairing whole, and the two equations it provides contain the one already supplied by the two edges. The count comes out the same, with a square and a circle on a face being alternative forms of one new fact: the metric on that face.

At ordinary precision #

An exact closure is one thing, and the sharp question is what the reading is worth from marks of the kind a photograph has. Take six edges to each direction, each 160 px long, read to half a pixel at both ends.

The square reads a true 1.33 to $\pm 0.021$. A second view of the same box, turned by 62°, reads it to $\pm 0.012$, with no extra knowledge of the scene at all: a second view supplies three more right angles from the same inner camera. That counts as a different kind of extra fact, namely that the two pictures share a camera, and it wins by about a factor of two here. It also has a price of its own, since a zoom or refocus between the two frames would break the assumption without any visible sign, and anamorphics do breathe.

Assuming a column, five pixels out, gives a bias of 0.008. It looks better than the others on the chart. That comparison has to be read with care, because it leaves out two things: a real column error is often much larger than five pixels, and this bar has no reading error in it, since exact vanishing points were used to draw its slope. So it is included to show the kind of number a reasonable-looking assumption produces. It is not a measurement of a method.

What can be said with confidence is that a width of 1.33 is very different from a width of 1 at every level of precision shown here. That was never in doubt for a 2-to-1 lens. It is the milder squeezes that matter, since those are the ones the pixel that is not square called the dangerous size. A square on a face is enough to detect them.

The scatter falls from $\pm 0.075$ at three short edges to $\pm 0.021$ at six longer ones, $\pm 0.015$ at twelve, $\pm 0.010$ at twenty-four and $\pm 0.007$ at forty-eight. It follows the expected path: past the first step each doubling of the edges improves it by about the square root of two. A photograph of a building holds about a dozen usable edges in a direction, so one to two per cent is what a single picture supplies. The common anamorphic squeezes, 1.33, 1.5, 1.8 and 2, are well over that apart, so one such picture can tell which of them was on the camera.

What else the fourth equation returns #

The width is the number this reading is after, but the equation closes the whole family, and the other three numbers come back with it. At the same six edges a direction, the focal length reads to ±15 px and the centre to ±23 px in the column and ±12 px in the row. Those look large beside a width read to two per cent, and it is the same conditioning seen from another side: a centre read from vanishing points has always been the weakest thing about this recovery, with or without a free width, and it is why nobody should expect a picture of a box to locate a lens’s optical centre to a useful precision.

There is a practical point inside that. Fit all four numbers, and the width is well determined while the centre is not. Fix the centre at a reasonable guess, and the width takes on all of the centre’s error. So the choice between fitting and assuming comes out differently depending on which number the reader cares about: a reader who wants the width should fit the centre, however poorly it is read, because that spends its uncertainty where it costs least; while a reader who wants a good focal length and is confident that the pixel is square should fix the width, as nearly everyone already does.

A small upward bias #

Reading a width at many true values shows something the scatter alone does not.

1.007 at a true 1, 1.108 at 1.1, 1.209 at 1.2, 1.341 at 1.33, 1.513 at 1.5, 1.767 at 1.75 and 2.022 at 2. The offset is about half a bar at every format and it does not vanish with more trials: at 120 trials its own standard error is about a fifth of the size. So the method reads slightly high. A square-pixelled camera comes out about 1.007, seven parts in a thousand wide.

The cause is the ordinary one for fitting a line through points with error in both coordinates, in which reading error rotates each fitted edge a little and so pulls the vanishing point toward the frame on average. A far point is pulled by more, and which point is far depends on the squeeze. The bias is a real property of an unweighted line fit, and it does not come from the square. A better estimator, with edge endpoints treated as noisy in the correct way, should reduce it. As a consequence, a width read as 1.01 from an ordinary picture is no evidence of anything. A width read as 1.05 starts to be.

How the view affects it #

The square’s face has to be seen, and how well it can be seen depends on the turn. Sweeping the box’s yaw through the range for a square on the face it carries away from the camera gives a curve with one striking feature.

From $\pm 0.009$ at 10° it climbs through $\pm 0.021$ at 38° to $\pm 0.119$ at 75°, as the face goes towards edge-on and its diagonals become hard to read. That part is expected. The spike comes as a surprise: $\pm 0.240$ at 30°, about ten times its neighbours. At a turn of 29.2° with this camera’s pitch of 26°, one of the square’s diagonals is parallel to the picture plane, and its vanishing point has gone to infinity. The measurement fits a finite point from nearly parallel lines, which is a poor way to find a point at infinity, and that fit is what fails. It is not a failure of the geometry, for which a homogeneous fit would be the repair, and the same spike would appear at every turn where any direction involved goes parallel to the picture. A careful reader of such a picture would notice a set of edges that hardly converge, and would treat its reading as weak.

It is a further illustration of a point these recoveries keep making: the conditioning depends on the picture’s arrangement, and it can change rapidly as the arrangement passes through a special case. A photographer has no reason to avoid that arrangement, and the same effect for a shifted lens makes a level camera’s vertical vanishing point weak.

A rectangle taken for a square #

The real risk in the method is not noise. It is that the square is not one.

The camera in this figure has square pixels. The rectangle’s diagonals are not perpendicular, and the reading treats them as though they were, so the solver has to find a camera in which they are. It finds one, and the width it reports is 1.108 for a 1.1-to-1 rectangle, 1.415 for a 4-to-3 and 1.709 for a 3-to-2. Exact marks, zero residual and no warning. To first order the reported width is just the rectangle’s ratio, and that makes sense: stretching a rectangle’s picture along one axis is how to turn it into a picture of a square, and the width is the stretch that does that. The focal length and principal point move as well, to 776 and (414, 131) at 1.1, and at 1.5 the focal length has fallen to 340.

That is the honest limit of the method, and it is a clean statement: a square on a face reads the pixel’s width only to within how well the square is known to be square. A window assumed square that is actually 10 per cent taller than wide reads as an anamorphic of ten per cent. So the method cannot tell a squeezed square from a true rectangle, because they are the same object in the picture: an anamorphic picture of a square actually is an unsqueezed picture of a rectangle. That is the earlier point about a face in another form. The viewer is relying on what a thing is known to be.

Therefore the strongest targets are things whose squareness is guaranteed by how they are made and which appear in numbers: a chessboard, a tiled floor, the printed faces of a calibration target. Two different assumed squares in one picture provide a check, because they will agree only if both really are square or both are out by the same ratio, and the second is rare. A window is far from the best choice.

A second view is usually available, and what it costs #

The comparison above favoured a second view, and that deserves a closing thought, because it is available in more places than it might seem. A film has many frames. Over a sequence with a camera move, every frame showing a corner gives three more right angles from one inner camera. A single anamorphic shot with a pan across a building, then, is rich in information about the lens’s squeeze, needs no square at all, and costs no assumption except that the zoom and focus are not changing and the camera’s inner numbers are still. For fixed-lens cinema that is often the case.

Whichever route is taken, an honest report gives its reading with its assumption attached. A wide field on a small screen shows how a viewer’s seat changes what a stretched picture looks like, and there is no point reading a squeeze to a per cent for display purposes if the picture will be seen from the wrong seat. For a reading used in reconstruction or for matching a model to a plate, it matters very much, because every other number the camera supplies depends on it.

Distortion, skew and edges cleaner than half a pixel #

Distortion: a wide or anamorphic lens bends the edges, which breaks the straight-line fit, and anamorphic lenses often distort differently along the two axes, so that the squeeze is not one number across the frame. Treating that would take a joint fit of a kind not attempted here.

Skew: a further unknown would need a further equation, so a fifth right angle or a second square on another face. The linear solver extends directly, and the count of equations against unknowns is the same argument one step on.

The noise model: half a pixel per endpoint is a simple assumption, and real edge detection is much better along a long sharp edge and much worse on a soft one. The scatter quoted should be read as a scale rather than as a promise. With very clean edges the bias described above may become the larger of the two errors, which is why it has been reported rather than averaged away.

Still open: whether a picture of a circle reads a squeeze with no box #

A circle on the ground, level, named the column and focal length in a circle on the ground names the column on a square-pixelled assumption. Let the pixel go free and a single circle on a known plane supplies two numbers about the inner camera, with one horizon-line. It is a much cheaper target than a box and a square, since many pictures have a round thing on a level surface and no visible corner. Two circles on one plane say nothing more than the first, as two circles leave the eye on a circle found for the pitch. Circles on two different planes, on the other hand, say something different.

The measurement that settles what such pictures can read takes round objects on two planes at a known angle, wheels on a car against a manhole on the ground or a clock on a wall above a round table, and asks how many inner numbers come back with the pixel’s width free; how well the width in particular comes back at half a pixel; and what happens when one round thing is not quite a circle, as a wheel seen under a car’s wheel arch is not, and the fit is forced to absorb the difference as a squeeze.

A squeeze and a rectangle are the same picture. What reads a pixel’s width is not more careful looking at the page but a fact about the scene, stated openly, and the reading is only as good as that fact.