Split the track and the needles turn

The spread a point gets found that a camera track’s arc is the wrong number to describe it by. A track covering sixty degrees gives no point of the scene sixty degrees: the nearest points subtend eighty-nine, the furthest forty-one, and each point’s error follows the angle at the point as its −1.68 power. The size of a point’s error is set by how widely its own lines of sight are spread.

The essay ended by pointing out that the covariance each error comes from says more than its size. The error is a needle rather than a ball, long in one direction and short across it, and the needle’s direction is a second thing a track decides. Reconstructions are usually wanted for a purpose with a direction in it — a facade’s flatness is a measurement across one axis, a floor’s level across another — so it asked whether a track can be arranged to point the needles usefully, or whether they simply point along the mean line of sight everywhere, leaving where the photographer stands as the only control.

They do not simply point along the mean line of sight, and a track can turn them. The rule that says how is short, and it has a threshold in it at exactly a right angle.

Every error is a needle #

Every figure here uses the exact linearised covariance of a point triangulated from cameras whose poses are known, each mark read with half a pixel of error — the uncertainty that repeated noisy triangulations would scatter with, computed directly rather than sampled.

On the familiar sixty-degree arc, every point’s error ellipse is elongated, two to four times longer than it is wide, and every one of them points roughly back toward the track. The strokes beside them are each point’s mean line of sight — the average of the directions from the cameras that see it — and the needles lie near them, within 8.4°.

That is the premise the earlier essay proposed, and to the eye it holds. It is also the case that makes the premise look inevitable. On one arc, all the cameras lie in roughly one direction from every point, so the direction the rays least constrain — depth along them — is the direction back toward the cameras. The interesting question is whether that is a law or a consequence of this arrangement.

Not the mean: the nearer rays count for more #

The needle’s direction has a precise description, and it is not the mean line of sight. Each camera constrains a point in the two directions across its ray, and it constrains them in proportion to how large its pixels are at the point: a camera twice as near puts pixels half as large on the point and says four times as much about where it is. The direction left least constrained is the one most nearly along the rays, with each ray counted by the square of its camera’s closeness — the principal axis of the lines of sight, weighted.

Tested against every point of the courtyard, the weighted axis is the better description by a wide margin: it misses the needle by a median of 0.36° and at worst 3.3°, where the plain mean misses by a median of 2.1° and up to 8.4°. The remainder is the part of each camera’s constraint that depends on where the point falls in its picture rather than how near it is, which a point near the edge of a frame feels and one in the middle does not.

The weighting is easiest to see with two cameras. The figure below holds two cameras thirty degrees either side of a point, one six metres away, and moves the other nearer and farther.

With both at six metres the needle lies on the bisector of the two rays, as it must by symmetry. Bring the second camera to three metres and the needle turns 23.1° toward it; send it to eighteen and the needle turns 27.1° the other way, toward the camera that is now the nearer. The mean of the two lines of sight does not move at all, since it depends only on their directions. The weighted axis follows the needle exactly. A camera’s say in where a point is grows as its pixels shrink at the point, and the needle leans toward the rays that say most.

At a right angle, the needle turns #

The weighted axis has a consequence that the single-arc plan hides completely. The needle points along the direction most nearly along the rays. If all the rays come from one side, that is back toward them. But if the rays come from two directions far enough apart, the direction most nearly along both of them is not between them at all.

Two cameras each twenty degrees either side of a facade’s normal, eight metres from a point on it, leave the point’s error at 11.2 mm along the normal — its depth into the facade — and 4.1 mm across. The needle points into the facade, back between the cameras. Open the two cameras to forty-five degrees either side and the two errors become equal: the ellipse in that plane is a circle. Open them further and the needle has turned by exactly ninety degrees, to lie across the facade: at sixty degrees either side, the depth is known to 4.4 mm and the position across to 7.6. The turn happens between 44° and 46°, all at once, because at the right angle there is no preferred direction and on either side of it there is one.

The two curves in the figure are mirror images about forty-five degrees. Two rays constrain a point everywhere across themselves; what they leave free is the one direction that is closest to lying along both, and for two rays symmetric about a normal that is the normal when they are less than a right angle apart and the direction across it when they are more. The needle is not a property of the direction toward the cameras. It is a property of the angle between them.

Two rays that do not meet found that two rays’ common perpendicular is where a triangulation hedges its bets; this is the same pair asked a different question — not where the point is, but which way it could slip.

What six pictures can do for a facade #

The threshold makes the earlier essay’s question practical. A survey of a facade’s flatness needs the depth of every point on it, along the normal, and cares much less about where each point sits along the wall. A single arc of cameras in front of it points every needle into the facade, which is the worst direction for that survey. The figure below takes six cameras and arranges them four ways.

Six pictures bunched on a narrow arc of thirty degrees measure the wall’s depth to 12.47 mm and its position along the wall to 3.42. Spread along one arc of ninety degrees they measure depth to 4.47 and along to 2.42 — better in both, since every point now gets a wider angle. Split into two groups of three at thirty degrees either side, 4.55 and 2.40, essentially the wide arc again. Split into two groups at sixty degrees either side, 2.83 mm in depth and 4.12 along.

The last arrangement measures depth four and a half times better than the narrow arc, from the same six pictures, and 1.6 times better than the wide one; it pays for it along the wall, where it is the worst of the four. It is the only arrangement in which the two groups of rays are more than a right angle apart at every point on the wall, and so the only one that turns every needle across the facade. Another picture of the same sweep found that what a reconstruction is short of is angular spread, not photographs; this says what that spread should look like when the reconstruction has a job: two groups, far apart, rather than an even arc.

So the arc’s shape matters, and not only its width #

The earlier essay’s alternative was that the needle points along the mean line of sight everywhere, in which case the arc’s shape would not matter and only its extent and position would. The measurements refuse that in two ways.

The needle follows the lines of sight weighted by closeness, so a track that brings some cameras nearer than others tilts every needle toward them. That is a control a photographer has without moving the track’s middle: walk in close on one side. And the needle follows the principal axis of those weighted rays, not their average, so a track whose cameras are gathered at its two ends points needles differently from one whose cameras are spread evenly across the same extent. A ninety-degree arc of six cameras and two groups of three at its two ends cover the same angle; the first leaves the facade’s depth at 4.47 mm, the second at 2.83. The shape is doing the work.

What does not change is that the needle can only point somewhere inside the fan of weighted rays, or square across it. A track cannot make a point’s needle lie in a direction its cameras do not span. A third ray is worth what its picture is worth found that each additional camera improves a point only as far as its own picture allows; here each additional camera also votes on the needle’s direction, with a weight set by its nearness, and the vote is over the directions the cameras supply.

Why a flatness survey is photographed from the sides #

There is a practical rule in the facade figure that experienced surveyors follow without this derivation: photograph a flat surface from well to either side, not from in front of it. The derivation says why and says how far. The rays from the two sides must be more than a right angle apart at every point being measured, which for a wall means more than forty-five degrees either side of its normal at every point, measured from that point. Near the ends of a long wall the angles differ, so the cameras have to stand wider for the far end than for the near.

It also says what the rule costs. The same pictures that fix depth well fix position along the wall badly, because the needle, having turned, now lies along the wall. A survey that needs both — the depth of a facade and the spacing of its windows — needs both kinds of camera, a group at each side for the depth and some near the front for the spacing, and the covariance says exactly what each group contributes.

The spread a point gets found that the size of each point’s error is decided by the angle at the point. The direction is decided by the same rays, read differently: not by how widely they spread but by how their weight is distributed across the spread. A track has two numbers to set for every point, and the second is its shape.

A floor asks for the same turn, stood on its side #

Nothing in the flip figure depends on the facade being vertical. It is a statement about two groups of rays and the normal between them, and it applies unchanged to any surface whose error along one direction is what a survey wants. A floor’s level is its error along the vertical; a floor photographed by cameras at eye height, all looking down at it from one side, leaves every point’s needle pointing back up along the lines of sight — steeply, toward the cameras — and the vertical is then the badly known direction.

The same rule says what would turn those needles off the vertical: rays arriving from two directions more than a right angle apart in the vertical plane, which for a floor means from low down on opposite sides, each more than forty-five degrees from straight overhead. A single camera held high over the middle does the opposite, pointing its point’s needle straight down. So a floor’s level is measured best by cameras that look along it from its edges, and worst by the overhead view that shows it most clearly — which is the facade’s lesson turned through a right angle, and the reason a surveyor’s level sights along the ground rather than down at it.

The general form is worth stating once. Whatever direction a survey needs a point’s error to be small in, the cameras should be arranged so that direction lies across the principal axis of their weighted rays — which, for two groups, means placing them more than a right angle apart on either side of it. The track and the scene, together solved for cameras and points at once without asking which way the points’ errors lay; the rule here is the question it did not ask.

What a needle is not #

One confusion is easy to fall into and worth excluding. Seven numbers no picture can name and an uncertainty is quoted from something found that a reconstruction’s uncertainties depend on what is held fixed — the frame the error is measured in — and that the same pictures can be given very different-looking error ellipses by holding different things still. The needles here are not that effect. The cameras are fixed and known, so there is no gauge left to choose, and each needle is the uncertainty of one point given the cameras: a property of the rays alone, measured in the world’s own frame. When the cameras are estimated too, the gauge returns and the needles acquire a part that depends on it, which is the question the last section leaves open.

What the covariances assume #

The cameras’ poses are known exactly. Every needle here belongs to a point triangulated from cameras that are not themselves uncertain. In a bundle adjustment the cameras are estimated from the same marks, and their uncertainty adds to each point’s in directions set by the whole track — a survey is trusted at its own accuracy, unless its error has a shape is where that part lives.

Marks are read with round, independent errors. A matcher’s errors on a facade are larger across repeated texture than along it, which gives each camera’s own constraint a direction and would tilt every needle further.

Every point is in view of every camera it is triangulated from. At sixty degrees to either side, a real facade’s reliefs hide parts of each other, and the points that only the front cameras see keep their needles pointing inward.

Still open: whether the bundle turns the needles back #

Everything above holds the cameras fixed and asks what the rays alone do to each point’s error. In a real reconstruction the cameras are found from the same pictures, and their errors are shared by every point they see. Two groups of cameras far apart are each well determined in themselves and poorly determined against each other, since few marks are seen from both sides, and an error in the relative placement of the two groups moves every point on the facade together — in a direction that is, for two groups either side of a wall, along its normal.

That would put back exactly the error the split was meant to remove, and the question with a number in it is how much. The measurement that settles it runs a full bundle adjustment on the six pictures of the facade in each of the four arrangements, computes each point’s covariance with the cameras’ uncertainty included, and asks whether the two groups at sixty degrees either side still measure the wall’s depth four times better than the narrow arc — or whether, once the cameras must be found too, the groups’ uncertainty against each other brings the needles back into the wall.