A turn of the head is not a step sideways

The last essay of the lens field ended on a quantity that was a nuisance. A camera swung on a tripod head does not rotate about its entrance pupil unless the head has been set up to, and the residual offset produces parallax that ruins a stitched panorama — a few millimetres of it, measurable in the seams.

This field wants exactly that offset. A few millimetres of translation between two views is a baseline, and a baseline is what makes a reconstruction possible. The same quantity, read as a signal rather than as an error.

The obvious question is how small it can be. The textbook answer is that a short baseline makes the problem ill-conditioned, which is true and is not what the measurement shows.

Which millimetres are being talked about #

One clarification before the measurements, because the quantity has a subtlety that the lens field had to work out and that would otherwise import an error here.

The baseline is the distance between the two cameras’ centres of projection, and a camera’s centre of projection is not the front of the lens, not the sensor, and not the tripod screw. It is the entrance pupil — the point that all the rays appear to pass through, which for a compound lens sits somewhere inside the barrel and moves when the lens is zoomed or focused.

That is why a panoramic head has to be adjusted rather than merely tightened, and why the adjustment is different for every lens. Rotating about a point a centimetre behind the entrance pupil is not a pure rotation; it is a rotation plus a small translation, and the translation is the parallax.

For this essay the consequence is direct. When a photographer swings a camera on an ordinary tripod, the resulting baseline is the distance the entrance pupil moved, which depends on how far the pupil sits from the axis of rotation. It can be a couple of millimetres on a well-set-up head or several centimetres on a badly set-up one, and the difference between those two cases spans most of the range the figure sweeps.

So “a turn of the head” is not a fixed configuration. It is a family of them, and where a given photographer lands in that family is decided by a mechanical detail they may never have measured — which is the lens field’s own finding about a quantity usually filled in from a manual, arriving here as the independent variable.

The degenerate case, exactly #

Start at the limit, where the situation is clean.

If the two views are taken from precisely the same point — a pure rotation, no translation at all — then the two pictures are related by a homography. Every point in one maps to a definite point in the other, by a single 3×3 matrix, regardless of how far away it is. That is what makes a panorama possible: a pure rotation destroys no information about the world’s directions and adds none about its depths.

In that case there is no reconstruction. Depth cannot be recovered because depth left no trace: two points at different distances along the same ray image to the same place in both pictures. The epipolar constraint has nothing to say, because there is no epipole; the essential matrix is identically zero; and the machinery of this field does not merely become imprecise, it becomes vacuous.

The routines here refuse rather than returning something. A triangulation of two rays through the same origin asserts and fails, with a message about the rays being parallel, which is the right behaviour: there is no answer, and returning a number would be inventing one.

And the case just next to it #

Now move the second camera by two millimetres.

The pictures are no longer related by a homography. The rays from the two eyes to a given point are no longer collinear. There is a baseline, an epipole, a rank-two fundamental matrix, and a reconstruction. Everything the field needs is present.

And the reconstruction is exact. Given correspondences placed to full double precision, a two-millimetre baseline recovers the courtyard’s shape to 4.7 × 10⁻⁹ — nine digits — and every longer baseline does better, improving smoothly to 10⁻¹⁴ at two metres.

That result is worth pausing on, because it contradicts the usual gloss. The algebra does not degrade at short baselines. It is not straining, or approaching a singularity that begins to bite at a centimetre. It works, and it works well, right down to the point where the baseline is exactly zero and it stops working entirely.

The design matrix’s spectrum says why, and says it quantitatively. The second-smallest singular value is proportional to the baseline over three decades. At two millimetres it is small — around 3 × 10⁻⁵ of the largest — but it is not zero, and a system whose smallest determined direction is 3 × 10⁻⁵ is perfectly solvable in double precision, which carries sixteen digits.

What actually fails #

Read the same marks to a whole pixel and the picture changes completely.

At a two-millimetre baseline the recovered shape is off by a factor of six billion more than the exact recovery. It is not a reconstruction of anything. And — this is the part that matters — it does not refuse. No assertion fires, no matrix is singular, no residual is enormous. A shape comes back, with a pose and a point cloud, and it is wrong.

The mechanism is the ratio between two quantities that the algebra never compares. The information about depth in the pair is proportional to the baseline. The error in the marks is fixed by how they were read. When the first falls below the second, the recovery is fitting rounding, and there is nothing in the fitting to notice that.

So the honest statement of the degeneracy is not “a short baseline is ill-conditioned”. It is:

The pure-rotation degeneracy is a fact about the measurement, not about the algebra. What matters is the baseline divided by the reading error, and a computation that does not know the reading error cannot report it.

Why the failure is silent #

It is worth being specific about why nothing complains, because the silence is the practical hazard.

The eight-point system is solved for its smallest singular direction. At a short baseline that direction is genuinely small — but so is the next one up, and the fit returns whichever direction the data points at. With noisy marks, that direction is determined by the noise rather than by the geometry, and it is still a unit vector, still reshapes into a matrix, still has a rank enforced onto it, still has a null space, and still produces an epipole and a pencil of lines.

Every downstream stage then works perfectly on a wrong input. The essential matrix decomposes into four poses. Cheirality picks one. Triangulation returns points. The points reproject to within a fraction of a pixel of the marks, because the marks are what the geometry was fitted to.

That last point is the sting. The reprojection error is small precisely because the fit used the marks. A reconstruction fitted to noise explains the noise well. Reprojection error is a measure of internal consistency, and internal consistency is exactly what a fit to garbage achieves.

A residual that cannot distinguish a good reconstruction from a fitted-to-noise one is another instance of the pattern this site keeps recording. The plumb-line fit returned 0.68 px with a distortion coefficient 22% wrong. The triangulation gap is invisible to reprojection error by construction. Here a whole reconstruction is wrong and reprojects beautifully.

The nearly-degenerate case has a shape #

There is a way of seeing what goes wrong that is more useful than “the system is ill-conditioned”, and it comes from asking what the fit is uncertain about.

At a very short baseline the two pictures are nearly related by a homography. Not exactly — a homography cannot account for parallax, and there is a little parallax — but nearly. So the space of fundamental matrices that almost explain the correspondences is not a ball around the right answer. It is a whole family, three-dimensional rather than one-dimensional, of matrices of the form $[\mathbf{a}]_\times H$ for the near-homography $H$ and any vector $\mathbf{a}$.

That family is what the null space collapses onto as the baseline goes to zero, and near the limit the fit is choosing between members of it almost at random. Each member has a different epipole, so the recovered epipole wanders over the whole picture between one noisy run and the next — and each member implies a different direction for the baseline, which is to say a different direction for the reconstruction to be stretched in.

The resulting reconstructions are therefore not merely noisy. They are systematically wrong in a way that looks structured: the point cloud comes out sheared or flattened along some arbitrary direction, which reads as a real shape rather than as noise, and a reader inspecting it will see a plausible if oddly proportioned courtyard.

That is worth knowing because it changes what to look for. Noise looks like noise and gets noticed. A confident reconstruction whose proportions are wrong along one axis looks like a reconstruction, and the axis it is wrong along is not the same axis from one run to the next.

The same structure explains the figure’s exact-input curve. With no noise at all, the fit still has to choose among that near-degenerate family — but the correct member is the one that fits exactly, and every other member has a residual, however tiny. Double precision is enough to see the difference down to a two-millimetre baseline, so the right answer is picked. Introduce a pixel of error and the residual differences within the family drop below it, and the choice is made by rounding.

What to look at instead #

Three quantities are available and none of them is the reprojection error.

The second-smallest singular value of the design matrix, which is proportional to the baseline and is computable from the two pictures alone. It does not need to be calibrated into metres to be useful — it says whether this pair has ten times the baseline of that one, and whether it is near the value at which the reading error takes over.

The angle at which the rays cross, at the depths of interest. That is the parallax angle, and comparing it against the angular precision of the marks is the range calculation applied point by point rather than to the rig as a whole.

And the gap between the rays, which is a length in the world along the direction the pictures are worst at. It grows as the crossing angle shrinks, which is exactly when the reprojection error stops being informative.

All three are cheap, and none of them is standard output.

The two-decade rule, and why it is not one #

It is natural to want a threshold — a baseline-to-range ratio below which a pair should not be trusted. The figure supplies the ingredients and deliberately does not supply the number, and the reason is worth stating.

The quantity that matters is the parallax angle compared with the angular precision of the marks, and both halves vary across a single pair of pictures. A near object subtends a large parallax angle and is well determined; a far one in the same pair is not. A crisply textured corner is read to a quarter-pixel; a soft edge on a shaded wall is not read to a pixel. So a single threshold for the pair is a statement about neither.

What can be said, and is worth carrying, is the form of the dependence. The uncertainty along the view direction goes as the range divided by the baseline, times the range, times the angular precision — that is, quadratically in range and inversely in baseline. Two of those three are usually known before a photograph is taken.

The instinct to want a rule of thumb is the same instinct that produced the sixty-degree cone of vision, which this site measured and found to be a statement about the reader’s viewing distance dressed as a statement about the picture. A number that has been detached from the quantity it was derived from stops being checkable, and the people who inherit it cannot tell which cases it covers.

So the recommendation is the unsatisfying one: compute the parallax angle for the depths of interest, compare it with the mark precision that is actually being achieved, and accept that the answer differs across the frame.

Where this leaves the panorama #

The connection to the lens field is worth closing, because the two essays are about the same millimetres.

A panoramic stitch wants the offset to be zero. If it is, the frames are related by a homography, they compose exactly, and the seams are perfect. Any residual offset produces parallax — near objects shift relative to far ones between frames — and no homography reconciles them, so the seam has to be hidden rather than solved.

A reconstruction wants the offset to be large. If it is zero there is nothing to reconstruct; if it is a few millimetres there is something, and whether it is usable depends on how well the marks can be read.

Both statements are about the same physical quantity, and the two fields want it pushed in opposite directions. What is worth extracting is that the offset is a measurable thing rather than a nuisance term: the lens field recovers it from the parallax it leaves, and this field recovers a proportional quantity from a singular value. A photographer who cannot set up a tripod head correctly has, in the failure, a measurement.

The middle ground is the uncomfortable one, and it is where a handheld pair of photographs usually sits. A step of a few centimetres, marks read to a pixel or better: enough baseline to reconstruct, not enough to reconstruct well, and no warning from any standard diagnostic about which side of the line the result is on.

A note on what the figure had to abandon #

The figure originally claimed that short baselines fail outright — that the recovery refuses below some baseline once the marks are quantised. An earlier probe on a slightly different canvas did exactly that, and the claim was written from it.

On this figure’s canvas nothing refuses. All ten baselines return a reconstruction, and the shortest ones return nonsense.

The corrected claim is the stronger one, and it is the essay’s title. A refusal would be a good outcome: a routine that declines to answer has told the caller something. What actually happens is that the routine answers, confidently, with a shape that is six billion times further from the truth than the exact recovery, and every downstream check passes.

The lesson generalises past this field. A failure mode that manifests as a refusal is a failure mode that will be found in testing. One that manifests as a plausible wrong answer with clean diagnostics will not, and the only defence is to compute the quantity that distinguishes them and look at it — which means knowing, in advance, which quantity that is.

It is also a lesson about how a claim gets into a document. The original sentence was not invented; it was read off a measurement, on a canvas 30 pixels narrower, where the recovery genuinely did refuse at short baselines. Nothing about it was careless. It simply described a boundary that moves with the configuration, and it was written as though it described the method.

The repair that made it robust was to stop reporting whether the recovery failed and start reporting by how much — a ratio between two curves rather than a threshold on one. Ratios of that kind survive a change of canvas, a change of scene and a change of point count, and thresholds do not. That is the same correction the normalisation figure needed when its single factor inverted, and the same one the many-view field’s view-count figure needed when adding cameras turned out not to help. Three times in one phase, and each time the fix was to measure a trend where a point estimate had been quoted.