Four cameras fit, and one of them can see

The fundamental matrix relates two pictures. It says nothing about where the cameras were, because it does not know what the cameras were: a picture taken with a long lens and a picture taken with a short one from the same place have different fundamental matrices with the second camera, and the matrix cannot separate the lens from the geometry.

Dividing out the lens is one matrix multiplication. What it leaves is a camera pose, and — this is the essay — it leaves four of them.

Taking the calibration out #

The calibration matrix $K$ holds the focal length and the principal point. It maps a direction in the camera’s own frame onto the picture. Multiplying by its inverse undoes that, turning a mark in pixels into the direction the mark came from.

If the constraint $\mathbf{x}'^{\top} F \mathbf{x} = 0$ is rewritten in terms of directions rather than pixels, the matrix in the middle becomes

$$E = K'^{\top} F K,$$

which is the essential matrix. It is the same geometric statement with the instruments divided out, and it is a much more constrained object: seven degrees of freedom become five, because the two that went were the ones describing the lenses.

Concretely, $E$ factors as $E = [\mathbf{t}]_\times R$ — a rotation and a translation, which is exactly a relative pose. Five degrees of freedom: three for the rotation, three for the translation direction, minus one because the length is not determined.

That last subtraction is not a technicality and it does not go away. It is the scale ambiguity arriving in the pose rather than in the scene, and it is why every function in lib/stereo.js that returns a translation returns a unit vector and says so in its name.

Four poses #

Recovering $R$ and $\mathbf{t}$ from $E$ is a singular value decomposition and a small amount of assembly. The assembly produces two candidate rotations, and the translation direction is determined only up to sign, so there are four candidate poses.

All four satisfy the epipolar constraint exactly, for every correspondence. There is no residual to compare, no fit to prefer, no numerical tie-break. The algebra is finished and it has produced four answers.

The four have a clean interpretation. One of the two sign choices reverses the direction the second camera moved. One of the two rotations is the true one and the other is the true one composed with a half-turn about the baseline. Combining them gives: the correct pair; the correct pair with the second camera on the wrong side; the pair with the second camera upside down; and both wrong at once.

Three of those four put some or all of the scene behind a camera.

The assumption that decides #

A point is in front of a camera when its depth in that camera’s frame is positive. That is a physical fact about photography rather than a mathematical one: a camera records what is in front of it, and a scene point behind the lens does not appear in the picture.

The test is called cheirality — the handedness of the reconstruction — and applying it is straightforward. Take a candidate pose, triangulate every correspondence with it, and count how many of the resulting points have positive depth in both cameras.

The figure’s answer is a clean sweep: 44 for the correct pose and 0, 0, 0 for the others. Not a majority, not a plurality. Every point is in front of both eyes for exactly one candidate and behind at least one eye for every other.

That decisiveness is worth drawing rather than asserting, because a narrow win and a sweep are different situations. A narrow win would mean the correspondences were poor enough that triangulated depths were landing near zero, which is a warning about the whole reconstruction rather than about the choice. A sweep means the geometry is unambiguous and the choice was never in doubt.

What deserves emphasis is what kind of step this is. Everything up to here has been linear algebra applied to marks on paper. This is not. It is a statement about the world — that these photographs are photographs of something the cameras could see — and it is imported from outside the algebra because the algebra has run out.

The site has been here before, and it is the same shape each time. The Renaissance constructions leave a free parameter that no drawn line determines, and the parameter is the viewing distance, supplied by a decision rather than by the construction. The height from a single view is a ratio until somebody’s eye height is supplied. Here the geometry is complete up to a fourfold choice, and what closes it is that photographers point cameras at things.

The comparison that measured the wrong thing #

The figure originally asserted that no rejected candidate came within a degree of the true rotation, so that the rejected poses could be described as not being near-misses. The assertion fired, reporting a rejected candidate 0.0° from the truth.

Nothing was wrong. The four candidates are two rotations, each taken with two translation signs, so one of the three rejected poses shares the winner’s rotation exactly. Comparing rotations alone cannot distinguish “the same turn of the head, with the photographer having stepped the other way” from “the same camera”.

And that candidate is not a near-miss at all. It puts the entire scene behind the cameras — 0 points of 44 — because reversing the baseline while keeping the rotation reflects the whole reconstruction through the first camera’s centre.

The corrected comparison takes each candidate’s rotation error and its translation-direction error and reports the worse of the two. The nearest rejected candidate is then 180° away, which is the statement that was wanted, and the near-identical rotation is now something the caption points out rather than something the assertion trips over.

The general form is worth carrying: a pose is not a rotation. Comparing poses by their rotations is comparing a six-dimensional object by three of its coordinates, and two poses can agree perfectly in those three and describe opposite situations.

The gauge the reconstruction is written in #

One choice is made silently at this point and is worth surfacing, because it looks like an arbitrary convention and is in fact the first appearance of a theme that dominates the many-view field.

The recovered pose is relative. It says how the second camera stands with respect to the first, and it says nothing about where either of them stands with respect to anything else — because there is nothing else. Two photographs contain no reference to a world coordinate system, and inventing one would be inventing information.

So the reconstruction is written in the first camera’s own frame: camera one sits at the origin, looking along its own axis, with the identity for its rotation. That is not a simplification for convenience. It is the honest statement that the pictures determine relative geometry and nothing absolute, expressed by picking one of the equally valid frames and saying which.

Every quantity reported in this field is therefore a quantity that survives that choice. Rotation differences, translation directions, distance ratios. A figure quoting the position of a reconstructed point in metres from some origin would be quoting the convention rather than the measurement, and would change if the convention did.

The habit becomes essential later. When six cameras are recovered together, the same freedom is present and has grown: seven numbers rather than the six here, because a sequence of pictures does not fix a scale either. Comparing a recovered track against a true one without first removing that freedom reports metres of error on a perfect reconstruction, which is the trap that field opens with.

What the fourfold ambiguity is not #

Two misreadings are common enough to be worth ruling out, and both make the situation sound worse than it is.

It is not a failure of the data. More correspondences do not reduce four to one. Better correspondences do not either. All four candidates satisfy the epipolar constraint for every point exactly, so no amount of data distinguishes them, and a practitioner adding points in the hope of resolving it is adding points to a question the points cannot answer.

And it is not a symmetry of the world. The four candidates are not four scenes that look alike. Three of them are not scenes at all: they describe configurations in which the photographs could not have been taken, because the subject was behind the camera. The ambiguity lives entirely in the algebra’s inability to represent “in front”, which is a sign condition and not a polynomial one.

That is exactly why the resolution has to come from outside. Cheirality is not an extra equation that was overlooked; it is an inequality, and the machinery that produced the four candidates only speaks in equations. Adding it is not refining the estimate — it is answering a different kind of question with a different kind of statement.

There is a practical consequence worth naming. When correspondences are poor enough that triangulated depths scatter around zero, the cheirality counts stop being a clean sweep and start being a majority. At that point the choice is being made by a vote among noisy measurements, and it can go the wrong way — producing a reconstruction that is a reflection of the true one, with every point mirrored through the first camera’s centre. The failure is total rather than gradual, and it is invisible in the reprojection error, because the mirrored reconstruction reprojects exactly as well. The count itself is the only diagnostic, which is why the figure reports all four rather than only the winner.

A repeated singular value, and a wrong answer with no symptoms #

The decomposition of $E$ produced this phase’s most expensive bug, and it is worth setting out in full because the failure had no early warning at all.

The standard recipe writes $E = U\,\mathrm{diag}(1,1,0)\,V^{\top}$ and reads the two candidate rotations off as $U W V^{\top}$ and $U W^{\top} V^{\top}$ for a fixed matrix $W$. The recipe is correct. What it assumes, without saying so, is a choice that an eigen-solver has no reason to make.

An essential matrix has two equal singular values by construction. Repeated singular values mean $U$ and $V$ are not unique: any rotation applied to both of their leading two-dimensional blocks gives an equally valid decomposition. The pair of candidate rotations survives that freedom, which is why the recipe works. What does not survive is the sign of the third columns, which the zero singular value leaves entirely free — and flipping one of them makes both products come back with determinant $-1$, which is a reflection wearing a rotation’s shape.

The symptoms were as follows. The fundamental matrix was recovered from the correspondences to 3 × 10⁻¹¹. The essential matrix to 5 × 10⁻¹¹. Its singular values printed 0.7071, 0.7071 and 0.0000, and its departure from being a proper essential matrix printed 6.6 × 10⁻¹⁰. Every intermediate quantity was exact to eleven digits.

The recovered camera was 39.04° out.

Worse, it was intermittent. The identical code on the identical cameras with thirty-two correspondences was perfect and with thirty-eight was wrong, because what changed was which basis of the degenerate eigenspace the Jacobi sweeps happened to land on. A bug that depends on the number of input points, with every printed diagnostic clean, is not one that gets found by staring at diagnostics.

The repair is one line: negate a candidate rotation when its determinant is negative. It loses nothing, because with the sign flipped the two candidates come back as each other, so the list of four is the same list in a different order and cheirality still has all four to choose from. And the standing check that goes with it is that every candidate pose is asserted to be a rotation before it is returned, so the next time a decomposition produces a reflection it fails immediately rather than four calls away.

The tolerance that was absolute #

Underneath that sat a second bug of the same family, and it is the one worth remembering, because the mistake is available in any numerical code.

Computing the left singular vectors as $\mathbf{u}_j = A\mathbf{v}_j / \sigma_j$ works whenever $\sigma_j$ is not zero. For an essential matrix $\sigma_3$ is zero, so the third column is exactly the case the formula cannot reach, and the code guarded it with a test on whether the singular value exceeded 10⁻¹².

A matrix normalised to unit Frobenius norm has its largest singular value at about 0.707. An essential matrix recovered from correspondences lands its zero singular value at around 3 × 10⁻¹¹, which sails through a 10⁻¹² test. So the third left singular vector was computed as the normalised direction of a vector that was entirely rounding error. It came out unit length, pointing nowhere in particular, and not orthogonal to the other two — the determinant of $U$ came out at 0.762 rather than $\pm 1$.

An absolute tolerance on a singular value is a bug whenever the matrix has been normalised. The threshold has to be relative to $\sigma_1$, because that is the only scale in the problem. The corrected code also re-orthogonalises against the columns already fixed, which costs nothing and makes the failure impossible rather than unlikely.

Both bugs shared a property that makes them worth two sections rather than a footnote: the recovery kept working, kept returning plausible poses, and kept producing figures. Nothing was ever thrown until a triangulation four calls downstream reported a point at infinity with a coordinate of NaN — a message about triangulation, from a bug in a singular value decomposition, in the one column the caller most needed.

What is now in hand #

After the decomposition and the cheirality test, the pair is fully determined in the following sense.

The rotation of the second camera relative to the first is recovered exactly — 0.0° from the truth on exact correspondences, which means to the arithmetic’s own floor. The direction of the baseline is recovered exactly. The length of the baseline is not recovered at all, and no amount of further correspondences will recover it.

That is a great deal. It is enough to triangulate every correspondence into a three-dimensional point, and therefore enough to produce a reconstruction of the courtyard — a real one, with the right shape, from two photographs and nothing else.

What it is not is a measurement. The reconstruction it produces is a shape without a size, and the next essay is about how completely that is true: the same two pictures are fitted equally well by the courtyard and by a courtyard three and a half times larger, and nothing in the pictures prefers either.

It is worth pausing on how much of the pinhole’s structure has now been used, because the answer is nearly all of it. The essential matrix exists because a projection is a map through a centre. The four candidates exist because a rotation composed with a half-turn about the baseline is still a rotation. The cheirality test works because a camera images what is in front of it and nothing else. Take away the centre — put the cameras behind glass, or in water — and every one of those statements weakens, in the way the refraction field measured for a single view.

That connection is not idle. A picture taken through a plane-parallel slab preserves every vanishing point exactly and moves every finite point by up to 4.5 px, which means the epipolar geometry of a pair taken through a display case is very nearly right and the triangulated positions are not. The recoveries in this field split along the same seam as the single-view ones did, and for the same reason: what survives a slab is direction, and this field’s pose recovery reads directions while its triangulation reads positions.