An uncalibrated picture ties the two groups, if the height of its centre is known

One square-on picture ties the two groups a third row only steadies found the cheapest repair for a flatness survey’s commonest failure. Six cameras in two groups, sixty degrees either side of a wall’s normal, every pose and mark found in one bundle adjustment, report the wall’s twist — a saddle, opposite corners pushed out together — to ±5.05 millimetres, because the two groups can pitch in opposite senses and the marks they share barely notice. A third row of marks steadies that to ±2.31. One picture taken square to the wall, belonging to neither group, ties them: ±1.25, and a five-millimetre twist is seen ninety-eight times in a hundred instead of sixteen.

That essay assumed the extra picture came through the same calibrated lens as the other six. It ended on the case that assumption excludes: the square-on view taken with a phone from across the street, whose focal length and principal point the adjustment would have to find for itself. A focal length, it said, is exactly the kind of number that trades against a depth — so perhaps the picture would scale itself along with the counter-pitch and let the twist back in.

The focal length turns out to be harmless. The number that lets the twist back in is one the question did not name.

Three unknowns, one of them fatal #

The survey is the earlier essay’s: a wall 6 metres wide marked in two rows of five, two groups of three cameras at 55, 60 and 65 degrees either side of the normal, 8 metres from the wall’s middle, 50-degree lenses, every mark read to half a pixel, seven numbers held for the gauge. The extra picture is taken from the same distance, ten degrees off square rather than exactly square — the reason for the ten degrees is the subject of a later section — and with the same lens, whose numbers the adjustment is now told it does not know. They are three: the focal length, and the column and the height of the principal point, the point where the lens’s axis meets the picture.

With its lens known, the picture ten degrees off square reports the twist to ±1.30 millimetres. With its focal length unknown, ±1.30 — the same to the third decimal place. With the column of its principal point unknown, ±1.30 again. With the height of its principal point unknown, ±4.83, barely better than the two groups alone. With all three unknown, ±4.83. Two of the three unknowns cost nothing at all, and the third costs the whole tie.

The focal length is free because the picture reads it, and reads it from something the twist does not touch. A camera ten degrees off square sees the wall’s far end smaller than its near end, and that perspective fixes the focal length against the camera’s distance to about 80 pixels. The twist moves the marks across the wall’s plane by a few millimetres and does nothing to that perspective, so the two are read from different features of the picture and do not trade. The focal length is read badly — 80 pixels is ten per cent — and it does not matter, because nothing the twist needs depends on it.

The height of the centre is a pitch with nothing to measure it against #

The adjustment’s covariance says why the height is different, in the same way it said what the twist was in the earlier essay: by what the twist moves with.

For a level picture the fitted twist and the height of its centre are correlated 0.96 at every turn from one degree to twenty, falling only to 0.78 by forty-five. The column of its centre is correlated with the twist not at all — zero to three decimal places at every turn. The earlier essay found the twist to be the two groups pitching in opposite senses, every camera’s pitch correlated with it at 0.94 or more. Moving a picture’s centre up or down shifts every mark it draws up or down by the same amount, which is very nearly what pitching the camera does; the difference between the two is a slight fanning of the marks towards the picture’s top and bottom edges. A level picture of a flat wall, seen nearly square, carries almost none of that fanning, so with its centre’s height unknown it cannot tell its own pitch from a shift — and a picture that does not know its own pitch cannot say which way the two groups are pitched either. The evidence that was to tie them is spent on the picture’s own lens.

The column is safe for the mirror-image reason. Moving the centre sideways is nearly what panning the camera does, but the twist does not involve a pan: the groups pitch against each other, they do not turn. A freedom the picture cannot settle is harmless when the quantity it trades against is not the one being measured.

The same correlation explains the remedy that works when the camera must stay where it is. A picture whose eye is raised to 3.5 metres and aimed down at the marks’ middle draws the wall foreshortened from top to bottom, and that foreshortening is what reads the height of its centre, as the turn about the vertical read its column. Square to the wall from that height, the correlation is 0.03; turned ten degrees as well it is 0.27, and at thirty degrees 0.46. Looking down at the wall separates the two numbers; turning sideways mixes them back a little.

Exactly square is a knife-edge #

The earlier essay’s picture was exactly square, and exactly square behaves differently.

With its lens known, the extra picture reports the twist to ±1.25 millimetres square to the wall, ±1.30 ten degrees off, ±1.46 at twenty and ±3.12 at forty-five, where it has nearly joined the group at sixty. With its focal length unknown the curve is the same, and with its centre’s column unknown it is the same again. With its centre’s height unknown it is ±4.82 at two degrees off square, ±4.83 at ten, ±4.87 at twenty and ±5.01 at forty-five — and ±1.25 at exactly zero.

At exactly zero the picture is symmetric about the wall’s middle, and the freedom an unknown centre adds becomes an exact trade with the camera’s own position: a picture square to a flat wall draws it identically if the camera stands a little higher and the centre is moved down to match. That trade moves the camera and no mark, so it costs the twist nothing. Half a degree off square the trade is no longer exact, the picture can almost but not quite tell its own height from its centre’s, and the twist leaks through the almost: what the picture can tell, it can tell only with the evidence the twist also needs. No photographer holds a phone square to a wall to half a degree, so the point at zero is not a position to rely on. It is the reason the earlier essay’s picture, which was exactly square by construction, could not have revealed what follows.

How well the centre must be known beforehand #

A phone’s principal point is not wholly unknown. A maker’s calibration places it, or an uncropped picture puts it within a few per cent of the frame’s middle, and the adjustment can be told that much as a reading of its own.

Known to a pixel or two, the centre is as good as a calibrated lens: ±1.30 millimetres. Known to five pixels, ±1.33; to ten, ±1.44; to twenty, ±1.79; to fifty, ±2.94; to a hundred, ±3.99. The scale is set by what the twist does to the picture. A millimetre of twist at the wall’s corners moves the marks two to three and a half millimetres across the wall, which from 8 metres is a fifth to a third of a pixel; to tie the groups the picture must hold its centre’s height to a few pixels, and a centre known only to fifty has given most of that away. The principal point is not the centre is the warning that goes with assuming the frame’s middle: a shift or a crop moves the centre by whatever was shifted or cropped, and a picture trimmed by a tenth of its width from one side has moved it thirty-five pixels from the new middle, outside what the tie can spare. An untouched picture from a phone whose maker states its centre is the case that keeps it.

A tape is not the missing number #

The question proposed two remedies, and they fare very differently.

A tape stretched between the two bottom corner marks and read to a millimetre leaves the twist at ±4.83 millimetres, exactly where it was. A tape fixes the survey’s scale, and the twist was never short of a scale: the scale is one of the seven numbers no picture can name, and it moves every mark along the wall’s own plane, which no flatness depends on. The missing number is the picture’s own pitch reference, and a length on the wall says nothing about it.

A second extra picture helps only if it shares the unknown. Two pictures ten degrees either side of square, each with a lens of its own, report the twist to ±4.74 — two pictures each unable to tell its pitch from its centre. The same two pictures through one lens, its three numbers unknown but shared, report ±1.19, the figure two calibrated pictures give. Each picture’s view of the wall fixes a different combination of the shared numbers, and between them they fix all three.

The strongest version of the same idea is the plainest. If the phone that takes the square-on picture also took the six oblique ones, its single unknown lens is shared by seven pictures, and the twist is reported to ±1.30 — as if the lens were known. The oblique pictures see the wall from sixty degrees, foreshortened in every direction, and they calibrate the lens far better than any near-square picture could; the square-on picture then has a known lens to tie the groups with. The failure is at the other extreme. Seven cameras each with a lens of its own report the twist to ±13.4 millimetres, two and a half times worse than the two groups did with lenses known, because every picture of a flat wall now carries a freedom it cannot settle.

One picture of a plane cannot calibrate itself #

Raising the eye read the height of the centre, and the obvious next step is to raise the eye with every number of the lens unknown.

With only the height unknown, raising the eye keeps the tie entirely: ±1.26 millimetres from an eye at 2.5 metres, ±1.29 at 3.5, ±1.37 at 5, the same as with the lens known. With the focal length and the whole centre unknown, it does not: ±4.74 at 2.5 metres, ±4.52 at 3.5, ±3.67 at 8. The arithmetic is the reason. A picture of a flat wall is a map from the wall’s plane to the picture’s, and such a map has eight numbers. A camera whose pose has six numbers and whose lens has three more has nine. One picture of a plane can settle eight of them, and the ninth is always free, however the camera is held. One conic calibrates the camera treats a lens as a conic in the picture; a picture of a plane supplies two conditions on that conic, and a lens with three unknown numbers needs three.

So the rule has two halves. A picture whose lens has one unknown number that it can read from its own view — the focal length from perspective, the column from a turn, the height from looking down — keeps the tie. A picture whose lens has all three unknown cannot keep it alone, and needs a second view of the same lens, or a prior on the centre, or a camera shared with pictures that see the wall at a steeper angle.

What a surveyor now sees #

The point of the earlier essay’s tie was what it let a survey see.

A five-millimetre twist is called real 97 times in 100 with the extra picture’s lens known, 93 with its centre known beforehand to ten pixels, and 17 with its lens unknown — one more than the two groups manage alone. A three-millimetre twist, 62, 53, 9 and 8. On a flat wall every arrangement calls a twist real five times in a hundred, as a test at twice its reported error should. A phone picture of unknown lens is not a tie; it is a sixty-first view of the same two directions.

That is the practical instruction, and it is short. Take the square-on picture with the camera that took the others, or take two square-on pictures with the phone, or know its principal point to ten pixels — which an uncropped picture from a phone with a maker’s calibration usually does. A picture from a different camera with nothing known of its lens adds nothing, and a tape on the wall does not rescue it.

What the linear model leans on #

Every number comes from the adjustment’s covariance at the true solution. That is the first-order description of a least-squares fit. Near the knife-edge at exactly square it is the first-order description of a nearly singular one, and there the reported errors are a statement about what the readings can tell apart rather than a forecast of what a solver started from a guess would find.

The extra picture’s lens is a pinhole with square pixels. A phone’s lens also bends straight lines, and its distortion is a further set of unknowns. The lines that calibrate a lens is about reading that distortion from straight edges; a wall’s string courses are such edges, and whether they would read it without eating into the tie was not measured.

The marks are matched correctly in every picture. A square-on view of a regular facade is the view most prone to matching a mark to its neighbour, and a wrong match is not a small error is the reminder that such a mismatch is a coherent displacement, not noise; the earlier essay’s residual test applies to the extra picture’s marks unchanged.

The groups’ lenses are known. The case of every camera uncalibrated with one shared lens kept the tie, and every camera with its own lens lost everything; the middle case — the groups calibrated, the extra picture’s lens shared with nothing — is the one measured here.

The general shape of it #

A bundle’s twist is tested by its own error showed that an adjustment’s reported errors can be trusted to say when a twist is real, and the earlier essay used that to show one picture’s worth. This essay finds the condition on that picture, and it is a familiar one in a new place. Split the track and the needles turn found that a survey’s weakness is whatever its views cannot hold against each other; where the adjustment stops found that every unknown the adjustment must find for itself can only add to the error. An unknown lens adds error only along the directions it can trade with, and the height of a level picture’s centre can trade with exactly the pitch the twist is made of. The focal length, which the question suspected, trades with the picture’s distance, which the twist does not care about.

Still open: whether a picture held level reads the height of its own centre #

A level picture’s principal point lies on its horizon. A level picture shows its rise on its horizon made that the basis of reading a shift lens’s rise, and it applies here: if the phone was held level — its verticals drawn parallel, which a wall’s edges and window jambs show — then the height of its centre is the height of the horizon, and the horizon is where the wall’s horizontal lines meet, which a picture turned ten degrees off square draws inside or near its frame.

The measurement that settles what that is worth adds to the adjustment the condition that the extra picture is level — its pitch and roll held at zero, as a check on its drawn verticals would license — with its focal length and whole centre unknown, and asks whether the height of the centre is then read from the horizon well enough to keep the tie; how near square the picture can be before the horizon’s vanishing point runs out of the frame and stops reading; and how level the picture must truly be, since a phone held two degrees nose-down puts its centre 26 pixels below its horizon, and that is the size of error the tie was found to tolerate.