A point is a line over there

A single photograph destroys depth. That is the first thing this site established and the thing every later field has been built against: a point in a picture does not say where a thing is, it says which ray the thing is on, and everything along that ray images to the same place.

The obvious hope for a second photograph is that it undoes the loss. It does, and the way it does is worth being careful about, because the naive version — two pictures, therefore two positions, therefore the answer — skips the step that carries all the content.

A point in one picture does not determine a point in the other. It determines a line.

Counting what is left #

The bookkeeping is the clearest way in.

A mark in the right-hand picture has two unknowns — its two coordinates. Before anything is known about the left picture, finding it means searching a two-dimensional region.

Now suppose the corresponding mark in the left picture is known, and suppose the epipolar geometry of the pair is known. The constraint $\mathbf{x}'^{\top} F \mathbf{x} = 0$ is one scalar equation. One equation removes one unknown. So the match is confined to a one-dimensional set — a line — and one unknown remains.

That remaining unknown is depth, and it has to remain. If the constraint pinned the match completely there would be nothing left to measure, and two pictures would contain no more information than one carefully labelled one. Where along the epipolar line the match falls is exactly how far away the point is, and it is what the second picture is for.

So the arithmetic is: two unknowns, minus one constraint, leaves one, which is the quantity a single view destroyed. Two views restore precisely what one view lost, and no more.

Why it is a line #

Take a mark in the left picture. Everything that could have produced it lies on one ray from the left eye — that is the single-view statement, unchanged.

That ray is a set of points in the world. Each of them has an image in the right-hand picture. The image of a straight line under a projection is a straight line, so the images of all of them form a line in the right picture.

That is the whole derivation. The epipolar line of a mark is the image, in the other picture, of the ray the mark came along.

Two features of it are worth pulling out because they explain everything the figure shows.

Every epipolar line passes through the epipole. The ray starts at the left eye, so the left eye is on it, so the left eye’s image is on the ray’s image. The left eye’s image in the right-hand picture is the epipole. Every ray from the left eye shares that starting point, so every epipolar line shares that point.

And the line does not depend on the scene. It depends on the mark, the two cameras, and nothing else. There need not be anything at all along the ray; the line is still there, and it is still where the match would be if there were.

The figure measures both. The five drawn lines meet at the epipole to 3.5 × 10⁻¹¹ px, which is a statement about concurrency rather than about drawing — the lines are computed independently from five separate marks, and the fact that they have a common point is a consequence being confirmed rather than a construction being drawn.

The measurement that matters #

The figure’s headline number is 1.4 × 10⁻¹³ px, and it is the distance from each match to the line its partner predicted.

The distance is measured in pixels, symmetrically — the mark in the right picture to the line predicted from the left, and the mark in the left picture to the line predicted from the right, averaged. That choice is deliberate and is worth a paragraph, because the obvious alternative is worse in a way that is hard to see.

The obvious alternative is to report the algebraic residual $\mathbf{x}'^{\top} F \mathbf{x}$ directly. It is already computed, it is zero when the constraint holds, and it appears in every treatment of the subject. It is also in no unit at all, and it scales with $F$ — which is only defined up to scale — so it can be made as small as desired by dividing the matrix by a thousand. A figure quoting it would be quoting a number that means nothing and can be improved by doing nothing.

The point-to-line distance is a length in the picture. It cannot be rescaled away, it is comparable between figures, and a reader who wants to know whether a fit is good can compare it against the size of a mark.

This is the third time on this site that a residual has had to be re-expressed before it said anything. The conformality test gave the cylinder a perfect score until the right angle was rotated. The anamorph’s distortion read as 2.34 against 2.42 — “nearly the same shape” — for a drawing that is plainly unreadable, until it was measured as a local area scale and came out at 7.2. Each time the arithmetic was correct and the quantity was wrong.

Seven numbers, not nine #

The fundamental matrix has nine entries and seven degrees of freedom, and the gap between those two counts is not bookkeeping — both of the missing two have consequences that show up in the figures.

The first is scale. The constraint $\mathbf{x}'^{\top} F \mathbf{x} = 0$ is homogeneous, so multiplying $F$ by any non-zero number changes nothing. A matrix and a thousand times that matrix describe the same epipolar geometry, which is why any comparison between two fundamental matrices has to normalise them first, and why the algebraic residual is worthless as a measure.

Scale is not the whole story, though. Comparing $F$ with $-F$ is comparing the same geometry with itself, so a routine that normalises and then subtracts reports a perfect recovery as being maximally wrong. matrixDistance in lib/stereo.js takes the smaller of the two differences for exactly that reason, and the reason is written into the function rather than into a comment somewhere else, because the failure it prevents looks like a broken recovery rather than like a sign convention.

The second missing degree of freedom is rank. $F$ is not merely a matrix that happens to satisfy the constraint; it is singular, with determinant exactly zero. That is what gives it a null space, and the null space is the epipole.

The consequence is that a matrix fitted by least squares to nine unknowns is not yet a fundamental matrix. Real correspondences, read to finite precision, produce a matrix whose smallest singular value is small and not zero, and such a matrix has no null space, therefore no epipole, therefore a pencil of epipolar lines that do not quite meet.

That failure is a beautiful one to look at and an easy one to misdiagnose. Nothing throws. The lines are drawn, they are individually in almost the right place, and they miss a common point by a few pixels — which reads as a drawing error, or a rounding problem, or a slightly wrong camera. It is none of those. It is a nine-parameter answer to a seven-parameter question, and the repair is to force the smallest singular value to zero before anything else is done with the matrix.

fundamentalFromPoints does exactly that, and returns the un-enforced version alongside as Fraw together with the singular value it threw away, so that the size of the correction is visible rather than silently applied.

The search this makes possible #

The practical consequence is the one every treatment of stereo mentions first, and it deserves stating even though this site does not use it.

Matching a mark between two photographs, with no geometric knowledge, means comparing it against every position in the other picture. On a 690 × 400 frame that is 276,000 candidates. With the epipolar geometry known, the candidates lie on a line — at most 690 of them, and usually far fewer once the range of plausible depths is bounded.

That is a reduction of more than two orders of magnitude, and it is why the fundamental matrix is estimated first and the dense matching done afterwards in every practical pipeline. It is also, precisely, the boundary of what this site claims. The reduction is geometry and belongs here. What is done with the remaining line — how a patch is compared, how ambiguity along a low-texture edge is resolved, what happens where one camera can see a surface and the other cannot — is the subject of matching, and every figure in this field hands over its correspondences exactly so that the geometry is what is under test.

The separating question is simple enough to apply. Does the argument survive with no image data at all, only coordinates? Everything in this field does. A matching argument does not.

The special case where the lines are parallel #

The pencil of epipolar lines converges on the epipole, and the epipole is a projective point, so it is allowed to be at infinity.

When it is, the lines do not converge — they are parallel. That happens exactly when the baseline is parallel to both picture planes, which is the case of two cameras facing the same direction, side by side. It is the configuration a stereo camera is built in and the one a pair of eyes approximates, and it has a name: a rectified pair.

Its convenience is enormous. If the two picture planes are also aligned, the epipolar lines are not merely parallel but horizontal and at the same height in both pictures, so a mark at row 173 in the left picture has its match at row 173 in the right. The search collapses from a line at an arbitrary angle to a single row of pixels, and the difference in column between the two is the disparity — which is the whole of depth measurement in that configuration.

The routine that finds the epipole in lib/stereo.js returns it as a direction with a flag when the third homogeneous coordinate vanishes, rather than dividing through. That is not defensive coding for a case that never happens; it is the case a stereo rig is deliberately built to be in. A routine that divides unconditionally reports the epipole of a rectified pair at around 10¹⁶ and draws nothing, and the resulting empty figure looks like a rendering problem rather than a division by something near zero.

What the constraint does not do #

Three things, and each of them is a place where the constraint is routinely over-read.

It does not say the match exists. A point visible in one picture may be hidden in the other — behind a block, outside the frame, or facing away. The epipolar line is still perfectly well defined and there is simply nothing on it. The constraint is necessary and not sufficient, and no amount of care with the geometry changes that.

It does not say a point on the line is the match. Every point along that line in the right-hand picture satisfies the constraint, because the constraint is one equation and the line is its solution set. A wrong match that happens to lie on the correct epipolar line is invisible to the geometry, and it is the failure mode that survives every check made here. That is not a small class of errors either: a repeating pattern — railings, brickwork, a row of windows — produces candidates spaced along exactly the direction the constraint leaves free.

And it does not depend on the marks being right. $F$ is fitted to the correspondences supplied. Feed it a list containing a mismatch and it will return a matrix, with a null space, with an epipole, and epipolar lines that look entirely reasonable and are wrong for every point. There is no self-check inside the fitting, and the residual does not reliably name the culprit — which is the finding that essay is about.

Choosing the five marks #

A small note about the figure, because it recorded a mistake worth keeping.

The five marks were originally chosen by spreading them across the left-hand frame, which is right — five points clustered in one corner produce five lines at nearly the same angle and the pencil is invisible. But one of the five sat near the top of the left picture, and the line it determined in the right picture passed above the right-hand frame entirely.

The line was correct. It was simply not on the paper. So the figure drew four lines under a caption that said five, and every assertion passed, because each assertion was about the four lines that existed.

The repair is that the marks are now chosen from the points whose epipolar line actually crosses the right-hand panel, and the count is asserted. It is the same species of failure as the missing tick labels that went unnoticed across a whole phase: every gate this fleet has asks whether a label fits, contrasts, and stays inside the frame, and none asks whether it is there. The symptom is absence, and absence is what a checklist of properties cannot see.

The same constraint, read the other way #

Everything above has been written as though the left picture is where a mark is known and the right picture is where it is sought. The constraint has no such preference.

$\mathbf{x}'^{\top} F \mathbf{x} = 0$ is symmetric in the sense that matters: transposing it gives $\mathbf{x}^{\top} F^{\top} \mathbf{x}' = 0$, so a mark in the right picture determines a line in the left one via $F^{\top}$, with all the same properties. That second pencil converges on the other epipole, and the two pencils are in correspondence — the line through one mark in the left picture and the line through its match in the right picture come from the same plane through both eyes.

That plane is worth naming, because it is the object all of this is really about. Two eyes and any third point in the world define a plane, and a plane through both eyes cuts each picture in a line. Those two lines are the epipolar pair. Sweeping the third point around sweeps the plane around the baseline like the pages of a book, and each position of the plane picks out one line in each picture.

So the whole structure is a one-parameter family of planes hinged on the line between the eyes, and everything about matching between two pictures is the observation that a point and its match lie on the same page of that book.

Read that way, the two facts the figure measures stop being separate. The lines meet at the epipole because every page of the book contains the hinge. The match lies on the line because the point, its two images and both eyes all lie in one page. One picture, two consequences, and both of them measurable in pixels.

Where this leaves the pair #

The epipolar constraint is the useful content of the fundamental matrix, and it is a strong constraint — it removes half of what has to be searched, it is exact, and it holds for every point without reference to what is being photographed.

It is also, on its own, not a reconstruction. It says where matches must lie, not where anything is. Turning it into positions needs the calibration divided out, which converts the fundamental matrix into the essential one and the essential one into a camera pose — with four candidates and an assumption to choose between them — and then a triangulation, which turns out not to be an intersection.

Before any of that, though, there is the question of how $F$ was obtained from the marks in the first place. The answer is a linear system, and the basis it is written in changes the answer by more than an order of magnitude.