A start needs the sign of its depths, not their size

A narrow view keeps a second answer, inside out found that six pictures of a courtyard through a narrow field admit a second reconstruction: the scene turned inside out, near points far and far points near, which a bundle adjustment settles in when it is started there and which misfits exact marks by less than a pixel at a 3° field. A parallel projection cannot tell a scene from its mirror in depth at all; a pinhole can, by an amount proportional to the field of view, so the narrower the field the better the twin hides.

That essay started the adjustment exactly on the twin. No real pipeline does. A solver begins from an initialisation — typically a chain of pairwise reconstructions, as in the track and the scene together — whose depths are roughly right, or roughly wrong, or flattened by a poorly conditioned first pair into very little relief at all. The question a pipeline actually faces is which of those starts return to the scene and which fall into its twin.

The natural fear, stated at the end of the earlier essay, is that the answer depends on the field: that at a long focal length, where the twin fits nearly as well as the truth, the twin’s valley is also wider, and a slightly bad initialisation is enough to fall in. That is what this essay measures, and it is not what happens.

A straight path between the two answers #

A start is built as a blend. Every parameter of the reconstruction — every point’s coordinates, every camera’s position and orientation — is moved a stated fraction of the way from its true value to its value in the twin. At no fraction it is the true scene; at the whole way it is the twin exactly; in between it is a family of starts joining the two floors by the straightest path the parameters allow.

What the blend does to the scene is simple enough to draw.

The twin reflects every point’s depth about the middle of the scene, so a blended start scales every depth by one minus twice the fraction. A quarter of the way, the courtyard has half its true relief. Halfway, it has none — pressed flat against the plane through its middle. Three quarters of the way it has half its relief again, reversed. The five clouds in the figure are straight lines of slope 1, ½, 0, −½ and −1, exactly.

That gives the path a natural reading in terms a pipeline cares about. A start a fraction f of the way along it is a start whose depths have the true shape, scaled by 1 − 2f: the right sign and a reduced size below halfway, the wrong sign above it. An initialisation that has recovered depth badly — too little parallax, a noisy first pair — is a start near the flat middle of this path. An initialisation that has recovered depth inside out is a start near its far end.

Halfway is the one start whose scene gives no preference between the two answers, which makes it the natural guess for where the boundary between the two valleys runs. It is only a guess, because the cameras are part of the start too, and they are half-turned toward the twin’s cameras at the same time.

Where the boundary runs #

The measurement runs one full adjustment from each of seventeen starts between a third and two thirds of the way along the path, every mark exact, and records where each one settles: back at the true scene, in the twin, or in neither within a hundred and twenty iterations. It does that along the standing 60° arc of cameras and along a 20° arc, at a 3° field and at a wide one.

Every configuration returns to the scene from every start short of 42 per cent of the way, and falls into the twin from every start past 56 per cent. Those are the whole of the figure’s firm edges.

Between them the boundary is not a line. Along the 60° arc at a 3° field, the start 46 per cent of the way falls into the twin while the start at 48 per cent returns to the scene, and everything from 50 per cent on falls in. At 25° the pattern is the same two points earlier: 44 per cent falls in, 46 returns, 48 on falls in. Along the narrower 20° arc the interleaving is wider: at 3° starts fall in at 42, 46, 52 and 54 per cent and return at 44, 48, 50 and 56, and at 10° two of the starts in the band did not settle within the iteration budget at all.

So the valleys meet along a band rather than a ridge. Within it, starts two blend-points apart — a difference of four per cent in the start’s relief — settle in different answers, and which one a given start reaches depends on the details of the descent’s path rather than on how close the start is to either floor. That is the ordinary behaviour of a descent on a surface with two valleys and a flat saddle between them: near the saddle the steps are long and nearly sideways, and a step that happens to cross into one valley stays there.

The field does not widen the twin’s valley #

The finding the measurement was designed for is in the comparison between the rows, and it is a negative one.

At a 3° field along the 60° arc the twin misfits exact marks by 0.946 pixels. At 25° it misfits them by 7.528 — eight times worse, and a residual no one would accept. If the twin’s valley grew with how well the twin fits, the band would sit much further toward the truth at 3° than at 25°. It does not. It sits between 46 and 50 per cent at 3° and between 44 and 48 at 25°. If anything the twin reaches two blend-points further at the wide field, where it fits worse.

The same holds along the 20° arc: the band is wider there than along 60° at both fields, and at neither field does any start short of 42 per cent fall in.

The reason is that the field of view and the valley’s reach are properties of different parts of the error surface. The field decides how deep the twin’s floor is — how nearly the twin fits — because it decides how much a pinhole’s division by depth differs from a parallel projection’s discarding of it. The reach of the valley is decided by where the surface between the two floors turns, and along this path that is governed by the flat middle, where the start has no relief to be right or wrong about. The flat courtyard is equally far from both answers at every field, so the saddle sits near it at every field.

For a pipeline that is a reassuring statement with a sharp edge. The field of view does not make a narrow-field reconstruction more sensitive to a bad start. It only makes a bad outcome harder to recognise once reached.

Three kinds of second answer, and which this is #

A reconstruction problem can have more than one answer in three quite different ways, and the band belongs to only one of them.

The first kind is a continuous family: seven numbers no picture can name — where the scene is, how it is turned, how big it is — along which every picture is unchanged. A start anywhere in that family is already a solution, and no descent moves along it. It has no boundary, because nothing on it is wrong.

The second kind is a small discrete set that a test removes. Four cameras fit, and one of them can see found four camera pairs that reproject a pair’s marks exactly, three of which put the scene behind a camera; requiring the scene in front picks the right one, and a start near a wrong one is simply discarded rather than descended from.

The third kind is the twin: a genuine second valley, with a floor that fits nearly as well as the truth and no test that removes it, because every point of it is in front of every camera. Far enough away, a pair is one eye found the parallax a pair loses as its scene recedes; the twin is what the loss of perspective, rather than of parallax, leaves behind. Only this kind has a basin, and so only this kind has a question about starts.

That places the band. A continuous ambiguity cannot be fallen into and a discrete one with a test cannot be kept. A valley without a test is kept by whatever start reaches it, and the only protection is to start on the right side of the saddle between them — which along this path is the flat scene, and in the depths a pipeline recovers is the sign.

What an initialisation has to get right #

Read in terms of relief, the band says how good an initialisation’s depths have to be.

A start 42 per cent of the way along the path has its depths scaled by 0.16: the true shape, the right way round, at a sixth of its true size. Every configuration measured returns to the truth from there. A start 56 per cent of the way has its depths scaled by −0.12: an eighth of the true relief, reversed. Every configuration falls into the twin from there.

So what an initialisation needs to supply is the sign of its depths, not their size. A chain of pairs that recovers the courtyard with a sixth of its real relief, because its first pair was nearly degenerate, still leads the adjustment home. A chain that recovers even an eighth of the relief inside out leads it into the twin. What sits between is a band about a tenth of the true relief either side of flat, in which the outcome is not predictable from the start.

That is an easier requirement than it sounds and a harder one than it looks. Easier, because recovering which of two points is nearer is far more robust than recovering how much nearer. A chain and an adjustment found that a chain’s per-link errors wander rather than accumulate, and a wandering error rarely reverses a scene’s depth order wholesale. Harder, because a narrow field is exactly the arrangement in which a single pair of pictures recovers depth worst — the relief a narrow-baseline, long-lens pair returns is small and noisy — and that pushes the start toward the flat middle of the path, into the band.

How long a start near the saddle takes #

The descent itself says when a start was near the boundary, which gives a pipeline something it can see without knowing the answer.

Starts far from halfway settle in as few as 9 iterations. Near the band the count rises, to 20 at 46 per cent for the 3° field and 21 at 44 per cent for the 25° field, which are both starts that fell into the twin from inside the band. The rise is modest — a factor of two, not an order of magnitude — and it is not a reliable alarm on its own, since starts at the far end of the path, well inside the twin’s valley, take nearly as long at the wide field. But a reconstruction that took twice the usual iterations from a flat-looking initialisation is one that started near the saddle, and that is the reconstruction the earlier essay’s one-solve test — restart from the reflection and compare residuals — should be run on.

The settled answers themselves show no middle ground. Every start that settled reached one of two places: the scene, fitting exact marks to the arithmetic floor, or the twin, fitting them to 0.946 px at 3° and 7.528 px at 25°. None settled on a third reconstruction. On this path there are two valleys and a band between them, and nothing else — and a solver that has settled in either reports an uncertainty for that valley alone, since an uncertainty is quoted from something and a covariance is quoted from the floor it was computed at.

Why the depth is the thing to check #

The twin is worth a pipeline’s attention exactly where it fits well, and the earlier essay’s measurement of that is worth seeing beside the band.

The twin’s misfit falls in proportion to the field — 0.474 px at 1.5°, 7.528 at 25° along the 60° arc. The band above does not move with it. Put together, the two figures say that the probability of falling into the twin is about the same at every field, and the probability of noticing is not: at 25° a solver that lands in the twin reports seven and a half pixels on exact marks, and at 1.5° half a pixel — a residual that where the adjustment stops shows is entirely normal for marks read to a pixel.

So the check a narrow-field pipeline should make is not on the residual, which is exactly what the narrow field disarms, but on the start. An initialisation’s depth order can be read before any adjustment runs — by whether nearer surfaces occlude further ones, by which way the shading on a known surface faces, by which points the cameras see at larger scale as they approach — and a start whose relief is small enough to sit inside the band is a start whose outcome the adjustment should not be trusted to choose.

What one path through the parameters leaves out #

One path. The blend is the straightest path between the two answers, and a real initialisation is not on it: its errors are spread through every point and camera independently rather than being a scaled copy of the true shape. Whether a start whose relief is a sixth of the truth on average, with independent errors on top, behaves like the blended start at 42 per cent is not measured.

Cameras blended with the scene. Every start moves the cameras toward the twin’s cameras by the same fraction as it moves the points. An initialisation whose cameras are right and whose depths are flattened is a different start, and plausibly an easier one.

Seventeen starts per row. The band’s edges are resolved to two blend-points. The interleaving inside the band is measured, not characterised: whether it has finer structure at finer spacing is open.

Exact marks. With marks rounded to a pixel the twin’s floor rises and its margin over the truth narrows, which may move the band. It was not run.

The two unsettled starts. Along the 20° arc at a 10° field two starts inside the band had not settled after a hundred and twenty iterations. They are drawn as neither answer and not explained.

The band and the sign #

Blended part of the way from a courtyard toward its inside-out twin, a narrow-field bundle adjustment returns to the truth from every start short of 42 per cent of the way — a start with a sixth of the true relief, correctly signed — and falls into the twin from every start past 56. Between, starts two blend-points apart settle in different answers. The band sits in the same place at a 3° field, where the twin misfits exact marks by 0.946 px, as at 25°, where it misfits them by 7.528; a narrow field makes the twin harder to notice, not easier to reach.

Still open: whether a noisy start behaves like a flattened one #

The blend moves every parameter together, so a start on the path is the true reconstruction scaled in depth. A real initialisation is the true reconstruction scaled in depth and scattered: each point’s depth off by its own amount, each camera turned by its own small error.

The measurement that follows builds starts with a stated mean relief — the blend fraction — and a stated scatter of independent depth errors on top, and maps which answer each settles in across both. If scatter moves the band toward the truth, then the sign of the average depth is not enough and a noisy initialisation at a narrow field needs more relief than a clean one to be safe. If the band holds where it is under scatter, then the rule stated above is a rule about initialisations in general: get the depth order right on average, and the adjustment does the rest at any field.