A flat scene fixes no second eye

Eight points and the basis they are read in built the eight-point algorithm and measured what it costs to run it in the wrong coordinates. Everything in it assumed the marks were in general position, and the assumption was never stated because nothing in the arithmetic complains when it fails.

Put every mark on one plane and the algorithm does not complain then either. It returns a fundamental matrix, with a residual at the arithmetic floor, and the matrix is one of infinitely many.

What the design matrix loses #

Eight correspondences give eight linear conditions on the nine entries of F, so a nullspace of dimension one — the answer — and that is what having an answer means.

Marks on a plane give conditions that are not independent. The two pictures of a plane are related by a homography H, so b = Ha at every mark, and the epipolar condition bᵀFa = 0 becomes aᵀ(HᵀF)a = 0 — a quadratic form in a that vanishes at every mark. Any F whose HᵀF is antisymmetric satisfies it for every point of the plane, and the antisymmetric three-by-threes are a three-dimensional space.

So the nullspace has dimension three, not one, and the family of fundamental matrices fitting the marks has two free parameters after scale.

The diagnostic is the second smallest, not the smallest #

Worth stating plainly because the obvious instrument is the wrong one.

The smallest singular value always vanishes — that is the answer’s own null direction — so a ratio of the two smallest is a ratio of two nearly-zero numbers on a flat scene and reports nothing. What separates the two cases is whether the second smallest is also negligible against the scale of the problem.

Measured: on the flat scene it is 3.4 × 10⁻⁹ of the largest. On the same marks lifted off the plane it is 5.2 × 10⁻³. Six orders of magnitude, visible without knowing the answer, and visible to a solver that has only the photographs.

The family, exhibited #

A claim that a set of answers all fit is worth nothing until the alternatives are drawn, which is this collection’s standing rule about anything it says a picture leaves free.

Three members of the family are built from three different combinations of the nullspace’s three directions, each forced to rank two as a fundamental matrix must be. All three satisfy the epipolar constraint at every mark to 1.6 × 10⁻⁶ pixels. They are 1.41 apart after normalisation, which is most of the way to being orthogonal, and their epipoles land at three completely different places — one near the corner of the frame, one six hundred pixels off the canvas, one within a few pixels of the origin.

Drawn over the right-hand photograph, they give three different bundles of epipolar lines. Every mark lies on its own line in all three.

The control that makes it a measurement #

Three matrices fitting the data would look identical on a scene where the answer is unique, because three copies of the right answer also fit.

So the same three nullspace combinations are computed on the same marks lifted off the plane, and there they miss by 13.5 pixels. The family belongs to the plane, not to the arithmetic.

What the same photographs do determine #

Nothing has been lost. A different question is being answered, and it is answered better than the first one ever was.

The homography relating the two pictures of the plane is fitted from four of the marks and reproduces all twenty-four to 5.9 × 10⁻¹³ pixels. Fitted from the same four marks of the scene with depth in it, it is out by more than a pixel, because there is no such map.

So the pair of statements is symmetric and worth putting in one sentence: the flat scene determines the plane’s map exactly and the eyes not at all; the solid scene determines the eyes and has no plane map to determine. A degeneracy is a place where the question was the wrong one.

Why this matters more than it looks #

Planes are not a rare special case. They are most of what anybody photographs: a wall, a floor, a facade, a table, a painting, a page, a car park, the ground from an aircraft at altitude.

The failure is silent, which is what makes it dangerous rather than merely inconvenient. The solver returns a matrix; the residual is at the floor; every mark lies on its epipolar line; and the recovered pose is arbitrary. A turn of the head is not a step sideways records the same silence for a different degeneracy and draws the same conclusion — the algebra does not notice, and the thing that fails is the measurement.

The practical response is a model-selection step rather than a better solver: fit both an F and an H, compare how well each explains the marks, and take the winner. That is standard practice in structure-from-motion pipelines and it is this fact that makes it necessary.

The rank-two enforcement is not where it fails #

A detail worth having, because it is where a reader implementing this would look first.

A fundamental matrix must be singular, and the eight-point algorithm enforces that by taking the fitted matrix’s own singular value decomposition and zeroing the smallest. On a flat scene that step runs perfectly happily on every member of the family — each one is projected to the nearest rank-two matrix, and each one still satisfies every mark afterwards.

So the degeneracy is not a failure of the rank enforcement and cannot be caught there. It is upstream, in the design matrix, and the only place it shows is the spectrum.

What the three members disagree about #

They agree on every mark and disagree about everything else, and setting out exactly what “everything else” is makes the failure concrete.

Where the second eye is. The epipole is the image of one camera in the other’s picture, and the three members put it at three unrelated places. The image of the other eye is where this collection established what that mark means; here there are three of them and no way to choose.

Which direction the camera moved. The essential matrix decomposes into a rotation and a translation direction, and three different fundamental matrices give three different motions. A pipeline that fed any of them forward would produce a plausible camera track.

And where every point is. Triangulating the same marks against three different poses gives three different scenes, each of which reprojects onto the photographs exactly. That is the strongest form of the failure and it is what the surface two pictures cannot separate exhibits in a case where the ambiguity is finite rather than continuous.

What they agree on, besides the marks, is the plane. Every member of the family is consistent with the same homography, which is the one thing the evidence actually contains.

What a third picture buys #

Nothing, if the scene is still flat.

That is the sharpest way to state what kind of failure this is, and it is the subject of an ambiguity is not an uncertainty. More marks and more photographs are evidence about a quantity; here every member of the family predicts every mark in every picture of the plane, so there is nothing for the extra evidence to be about. The reconstruction does not get better slowly. It does not get better.

What buys the answer is motion out of the plane’s own degeneracy — a third camera not in the family’s own configuration, or a scene point off the plane. One point off the plane is enough in principle, and how flat is flat enough measures how far off it has to be before a reader’s own marking error can see it.

Where the plane’s own map is the better instrument anyway #

If the scene is a plane and stays a plane, the homography is not a consolation prize.

It gives the full relationship between the two pictures from four marks, exactly. It rectifies the plane, which is what flattening a facade does with it. It supports every metric measurement the plane can carry once one length on it is known. And it decomposes into a rotation, a translation and the plane’s own normal — up to a known two-fold ambiguity — so the camera pose is recoverable from a plane, by the route that knows it is looking at a plane.

The failure is therefore entirely a failure of using the wrong instrument, and the diagnosis is one number in a spectrum.

The other planar degeneracies, briefly #

The plane is the largest of a family of configurations that leave the epipolar geometry underdetermined, and it is worth knowing the others exist even though this essay measures only the one.

All the marks on a line is worse: two independent conditions instead of eight, and the fundamental matrix has six free parameters left. It is easy to spot and rare in practice.

All the marks and both camera centres on a ruled quadric is the interesting one, because it is not visibly degenerate at all — the scene has depth, the marks are spread over the frame, the spectrum looks healthy — and there are still two exactly correct answers. That is the surface two pictures cannot separate, and a plane through both camera centres is a member of that family, which is the relationship between the two essays.

And a scene far enough away makes the parallax fall below the marking error, which is a different kind of failure again: not an exact ambiguity but an uncertainty that grows without bound. Far enough away, a pair is one eye measures the crossing.

Why the algebra is worth doing rather than quoting #

The identity behind the whole essay is one line — bᵀFa = aᵀ(HᵀF)a, which vanishes for every a exactly when HᵀF is antisymmetric — and it is worth doing rather than quoting because it says which three dimensions the nullspace has.

An antisymmetric three-by-three is [e]ₓ for some vector e, so the family is F = H⁻ᵀ[e]ₓ with e free. That e is the epipole: every member of the family is a legitimate epipolar geometry for the plane, with the second eye placed anywhere at all. So the family is not a numerical artefact and not a near-degeneracy; it is a three-parameter set with a geometric meaning, and the meaning is the plane does not know where the second eye is.

Reading it that way also says what a single off-plane point buys, and the count is worked through in the section below. It comes to two — which is the arithmetic behind the practitioner’s rule of thumb that a nearly planar scene needs points off the plane, and it is a count rather than a feeling.

Two points off the plane, and the pair that are not enough #

The family is $F = H^{-\mathsf{T}}[\mathbf{e}]_\times$ with the epipole $\mathbf{e}$ free, and $F$ is projective, so the free part is a point of the projective plane: two degrees of freedom, which is what “a two-parameter family after scale” means. Counting how many off-plane marks it takes to remove them is then a matter of asking what condition one such mark imposes.

Take a correspondence $\mathbf{a} \leftrightarrow \mathbf{b}$ whose world point is not on the plane, so that $\mathbf{b} \neq H\mathbf{a}$. Its epipolar condition is $\mathbf{b}^{\mathsf{T}}F\mathbf{a} = 0$, and substituting the family gives

$$\mathbf{b}^{\mathsf{T}}H^{-\mathsf{T}}[\mathbf{e}]_\times\mathbf{a} = 0 \;\Longleftrightarrow\; \mathbf{e} \cdot \left(\mathbf{a} \times H^{-1}\mathbf{b}\right) = 0.$$

One linear condition on $\mathbf{e}$, and a homogeneous one. So one off-plane mark confines the epipole to a line in the picture, and a second, generically, to the intersection of two lines — which is a point. Two off-plane marks determine the epipolar geometry of a planar scene, and the family collapses in one step from a plane of possibilities to a single answer.

The condition has a reading that makes the count feel less like algebra. $H^{-1}\mathbf{b}$ is where the plane’s own map would have put the mark if its world point had been on the plane, so $\mathbf{a} \times H^{-1}\mathbf{b}$ is the line joining the mark to its plane-transferred position — the parallax segment the point’s height off the plane opens up. And the condition says the epipole lies on that line, which is a fact this collection already has in another field: parallax is radial about the epipole. Each off-plane point draws a line through the epipole; two lines meet at it.

That reading also names the configuration where two are not enough. The two lines fail to meet at a point when they are the same line — that is, when both off-plane marks and the epipole are collinear in the first picture. It is not a contrived case: a row of posts standing off a flat ground, photographed from a direction that puts them along the epipolar direction, is exactly it, and adding a third post on the same row buys nothing at all. The repair is a point off the plane and off that line, not a further point of the same kind.

So the honest form of the count is: two off-plane marks, provided they and the epipole are not collinear, and the second clause is unusually easy to violate because a scene’s off-plane features are so often arranged in a row. That is the same shape as the field’s other counts — eight marks in general position, four for a homography with no three collinear — and it has the same practical form: the number is small and the general-position clause is the part that fails.

One further consequence is worth stating, because it is the reason the diagnostic in this essay is a rank rather than a residual. Adding two off-plane marks to twenty-four planar ones does almost nothing to the residual, since twenty-four marks already fit perfectly and two more contribute two rows out of twenty-six. What it does is lift the design matrix’s second-smallest singular value off the floor. The evidence that the scene has stopped being degenerate is entirely in the spectrum, and a pipeline watching the residual would see a fit that was excellent before and is excellent now.

What the marks look like while this is happening #

Worth a paragraph, because the failure has no visual signature and a reader will want to know whether they could have spotted it.

The two photographs look like two photographs of a wall. The marks are spread across both frames, well distributed, sharply defined, and matched correctly — there is no outlier, no cluster, no degenerate-looking bunching. A feature detector is delighted. The residual after the fit is at the floor. Every quality indicator a pipeline computes says the data are good.

That is the whole difficulty and it is why the diagnostic has to be a rank rather than a residual. The evidence is not bad; there is simply less of it than the question needs, and “less than the question needs” is a statement about the question rather than about the marks. A picture of a wall is excellent evidence about the wall and no evidence at all about where the second camera was.

The short version #

Marks on one plane make the eight-point design matrix rank six, so its nullspace has three dimensions and a two-parameter family of fundamental matrices satisfies every mark. Three members of it fit to 1.6 × 10⁻⁶ pixels, sit 1.41 apart after normalisation, and place the second eye in three different parts of the picture; the same three combinations miss by 13.5 pixels on a scene with depth.

The diagnostic is the second smallest singular value against the largest — 3.4 × 10⁻⁹ flat, 5.2 × 10⁻³ solid — and it is visible without knowing the answer. What the same photographs do fix is the plane’s own map, from four marks, to 5.9 × 10⁻¹³ pixels.