The image of the other eye

Every essay on this site so far has had one camera in it. Sometimes the camera is stated and the picture is drawn from it; sometimes the picture is handed over and the camera is recovered from the drawn edges. Either way there is one eye, one projection, and one set of things that survive it.

This field has two. And the first thing worth knowing about two pictures of one scene is that each of them contains the other camera.

Not a picture of the other camera — the second photographer need not be standing in shot. A point. There is one point in the left-hand picture which is the image of the right-hand eye, and one point in the right-hand picture which is the image of the left-hand eye, and both of them can be found in the pictures without knowing anything at all about where either camera was.

That number is the whole essay. The rest is what it means and why it is worth a figure rather than a sentence.

Where the point comes from #

The eye that took the right-hand picture is a point in the world. Points in the world have images: that is what a camera does, and it does it to every point in front of it whether or not anything is there. So the right-hand eye has an image in the left-hand picture, in exactly the sense that a lamp post has one — a place on the picture plane where the ray from the left eye to the right eye crosses it.

That place has a name. It is the epipole, and there is one in each picture.

Reasoning about it directly is short and unhelpful. Every ray that leaves the left eye, whatever it is aimed at, lies in a plane with the two eyes. The image of that ray in the right-hand picture is therefore a line, and every one of those lines passes through the right-hand epipole — because every one of those planes contains the right-hand eye. The pencil of all such lines has one point in common, and the common point is where the other eye is.

None of that is hard, and all of it assumes what is being looked for. The interesting version is the reverse: given only a list of matched points, produce the epipole, and then check it against the camera position that was never supplied.

What the computation is given #

The correspondences are the whole input. Forty-four points in the courtyard are visible in both pictures, so there are forty-four pairs — a position in the left picture and a position in the right — and nothing else crosses into the computation. No focal length, no camera position, no orientation, no scene coordinates.

That restriction is enforced by a file boundary rather than by discipline. lib/stereo.js has a line in it, marked in the source, and everything below the line takes image coordinates. The two functions that see cameras sit above it, are labelled ground truth, and exist to be compared against. This is the same arrangement lib/projective.js has had since the single-view round trip was written, for the same reason: a recovery computed with the answer in scope is not a recovery, it is a redrawing, and the difference is invisible in the resulting picture.

The correspondences themselves are supplied by construction — one world point projected into two cameras. That is a deliberate limit on what this field is about, and it is worth stating early because it will keep coming up. Nothing here matches anything. Finding which mark in the left picture goes with which in the right is a large and genuinely difficult subject, and it is a different one; every argument in this field survives with every correspondence handed over exactly, which is the test that separates the geometry from the matching.

The matrix in the middle #

The route from a list of correspondences to the epipole runs through a 3×3 matrix.

For any pair of matched image points $\mathbf{x}$ and $\mathbf{x}'$, written as homogeneous three-vectors, there is a matrix $F$ with

$$\mathbf{x}'^{\top} F \mathbf{x} = 0$$

for every pair. It is called the fundamental matrix, and the constraint is one linear equation in the nine entries of $F$ per correspondence. Eight correspondences therefore determine $F$ up to scale, which is all that a homogeneous constraint can determine, and the name of the classical method — the eight-point algorithm — comes from that count.

What matters here is not how $F$ is found but what happens once it exists. $F$ has rank two, so it has a null vector: some $\mathbf{e}$ with $F\mathbf{e} = 0$. Feed that into the constraint and it says $\mathbf{x}'^{\top} F \mathbf{e} = 0$ for every $\mathbf{x}'$ — every point in the right-hand picture, without exception.

A point in the left picture whose constraint is satisfied by every point in the right picture is a point that says nothing about where its match is. There is exactly one such point, and it is the epipole. Similarly $F^{\top}$ has a null vector, and that one is the epipole in the right-hand picture.

So the epipole is not computed by a special procedure. It falls out of the null space of a matrix that was fitted to matched marks, and it is the direction in which the fitting has nothing left to say.

Why the check is the point #

Here is the arrangement the figure makes, in full.

Two cameras are built with stated positions. The courtyard is projected into both, producing forty-four pairs. The pairs are handed to a routine that has never seen a camera. It fits $F$, takes its null space, and returns two points.

Separately — and never in the same scope — each camera is asked to project the other camera’s eye, which is a world point like any other. That gives two more points.

They are the same points, to about a billionth of a pixel.

There is no way for that to happen by accident. The recovered epipole depends on forty-four pairs of numbers; the projected eye depends on two camera positions and two focal lengths. The only thing connecting them is that the correspondences were produced by those cameras, and the agreement says the epipolar geometry encoded in the marks really is the geometry of the eyes that made them.

The residual — a fraction of a billionth of a pixel — is arithmetic rather than geometry. It is the accumulated rounding of a least-squares fit, a rank enforcement and a null-space extraction in double precision. There is nothing left in it that is about perspective.

The matrix belongs to the cameras, not to the courtyard #

One property of $F$ is easy to state and easy to underrate: it does not depend on the scene at all.

The derivation makes this obvious in hindsight. $F$ is built out of the two cameras’ calibrations, their relative rotation and the direction of the baseline between them. Nothing about what is being photographed enters. Two photographs of a courtyard and two photographs, from the identical pair of positions, of a completely different courtyard have the same fundamental matrix.

The consequence is that the forty-four correspondences are not forty-four separate facts being averaged into a compromise. They are forty-four measurements of one object with seven degrees of freedom, and each of them constrains it in the same way. That is why eight suffice, and why the forty-fifth adds precision rather than complexity.

It also explains the shape of the failure when a correspondence is wrong. A mismatched pair is not a point that happens to be in an awkward place; it is a measurement of a different matrix, spliced into a list of measurements of this one. A least-squares fit has no way to notice that and no place to put it except spread across the whole answer, which is why one bad match damages every point rather than the region around itself.

And it is why the epipolar constraint is a genuine test rather than a restatement. Nothing in the fitting of $F$ used the fact that these particular marks came from a courtyard, so when a forty-fifth point drops onto the line its partner predicts, that is a fact about the cameras being confirmed by a point that took no part in establishing it.

The epipole is usually not on the picture #

Look at the figure again and notice that neither epipole is drawn as a dot on the courtyard. Both are arrows pointing off the edge.

At 2.6 m apart, with the cameras aimed at the same point in the courtyard, the right-hand eye is far outside the left camera’s field of view. Its image is therefore a point on the picture plane that lies well outside the picture frame — a perfectly good point, at coordinates that run into the thousands on a 690-pixel canvas.

This is the ordinary case, not a special one. Two photographs taken from positions a few metres apart, both aimed at the same subject, will almost always have their epipoles far off-frame, because seeing the other camera in shot requires pointing roughly at it. The exception is the rectified pair, where the two cameras face the same direction and the epipoles go off to infinity in the horizontal direction — a case common enough and useful enough to have its own field of arithmetic.

The habit worth carrying is that a projective point being off-canvas is not a failure. This site has met it before: the vertical vanishing point that the height recovery uses sits at y = 7.2 × 10⁹ on a 420-pixel canvas and the method is exact anyway, because it parameterises along a line rather than evaluating at a point. The routines here are written the same way — the epipole is returned as a direction with a flag when its third homogeneous coordinate vanishes, rather than divided through and reported at 10¹⁶.

What the epipole is good for #

Three things, in ascending order of usefulness.

It says whether two pictures are of the same scene from different places. If they are two frames from a camera that only turned, there is no baseline, no epipole in the ordinary sense, and the whole apparatus degenerates. That degeneracy has a measurable approach rather than a cliff, and a whole essay about why the failure is in the reading rather than in the algebra.

It organises the search for matches. Every point in the left picture has a line in the right picture that its match must lie on, and every one of those lines passes through the epipole. That turns a two-dimensional search into a one-dimensional one, which is the next essay and is the single most useful consequence the field has.

And it is the direction of the baseline, in the picture’s own coordinates. The line from one eye to the other images to a point, and that point is the direction the photographer moved, read off the photograph. Combined with a focal length it gives the direction of the second camera in the first camera’s frame — which is most of the pose, and the rest of it needs one further step and one assumption.

The camera, written as a matrix #

One piece of machinery had to be built before any of this, and it is worth a paragraph because everything in three fields rests on it.

This site’s camera is specified the way a photographer would specify one: an eye, a target, an up direction, and a focal length or a field of view. Multi-view geometry is written in terms of a 3×4 matrix $P = K[R \mid \mathbf{t}]$. Those are the same camera, and “the same” has to be a measurement rather than an assurance.

The rows of $R$ are the camera’s own basis vectors, because projecting a point in lib/camera.js is exactly three dot products with them. That includes the site’s down vector, which is forward × right rather than the other order — a sign that was wrong for a whole phase and imaged the ground above the horizon. A matrix written from those three vectors is right by construction. A matrix written from a remembered convention is right by luck, and the difference between the two shows up as a picture that looks plausible and is mirrored.

The measured agreement is 1.8 × 10⁻¹³ px over forty-four points from two cameras. That is the floor, and it means the matrix form introduces nothing.

What two pictures have not yet bought #

It is worth being precise about how little has been established.

The epipole is one point. It fixes the direction from one eye to the other, in the first camera’s frame, and it does not fix the distance. Nothing so far says how far apart the cameras were, how the second one was turned, or where anything in the courtyard is.

Those come next, and they come with qualifiers. The rotation and the direction of the baseline are recoverable exactly from correspondences and a focal length. The length of the baseline is not recoverable at all, from any number of correspondences, and neither is the size of the courtyard — which is the single-view scale ambiguity surviving the addition of a second camera entirely intact.

That survival is the thing this field keeps returning to. A second picture is a large addition: it turns ratios into a three-dimensional shape, turns a ray into a point, and puts the other camera on the paper. It does not turn a shape into a size. Two eyes 65 mm apart do not know they are 65 mm apart, and nothing they see will tell them.

The habit, restated for two cameras #

The site’s standing discipline is that every figure is projected from a stated camera and every recovery is checked against something it was not given. Doubling the number of cameras doubles the opportunities to cheat, and the two that matter are worth naming.

A reconstruction computed with the true poses in scope is a redrawing. It will reproject perfectly, it will look like a scene, and it will prove nothing. The defence is the file boundary described above, and it is checkable: the recovery functions take arrays of image coordinates, and there is no path from them to a world point.

And a comparison made without removing the freedom is meaningless. The reconstruction is only determined up to a similarity, so comparing recovered positions with true ones directly reports the arbitrary choice of frame as though it were error. Every comparison in these three fields removes that freedom first — pairwise distance ratios here, an aligned track in the many-view field — and the removal is done by a named function rather than by eye.

Both traps produce the same symptom, which is a number that is far too good or far too bad for reasons that have nothing to do with the geometry. The epipole check in this essay is designed to be immune to both: it compares two computations of one point, in pixels, in a frame both of them already live in.

Two pictures, forty-four matched marks, and the other camera falls out of them. That is where the field starts.