The picture contains what is behind the camera

A pinhole projection is written as a division: a world point at $(x, y, z)$ lands at $f x / z$ across the picture and $f y / z$ down it. The division is the whole of the perspective, and it has a property that is almost never stated because it looks like a triviality.

Change the sign of $x$, $y$ and $z$ together and the mark does not move. The numerator and the denominator both change sign and the quotient is what it was. A point and its exact reflection through the eye — the same distance along the same line, on the other side — have the same image.

That is not a defect in the formula and it is not a convention. It is the statement that a pinhole maps a direction rather than a point, and a line through the eye has one direction whichever way a reader walks along it.

The last sentence is the control and it is load-bearing. A figure showing that two things coincide is worth nothing until something has been shown that does not, and the obvious near-miss — reflecting through a point close to the eye — moves the mark half a hundred pixels. The coincidence is exact and it is about one point in the room.

What the ray picture shows that the algebra hides #

The formula makes the fact look like a cancellation. A section through the pinhole makes it look like what it is.

Read that way there is no coincidence at all. There is one line through the eye, it meets the image plane once, and the projection is the map from lines-through-the-eye to points-of-the-plane. Points of the world enter only by choosing the line they are on.

Which is why the set of directions, rather than the set of points, is the thing a camera actually photographs. Directions through a point in space form a projective plane, and every one of its members is a perfectly ordinary member — including the ones lying in the plane through the eye parallel to the picture, which have no image on the finite plane and which a line that closes treats as the point at infinity.

A photograph is a chart that misses one line #

Once the projection is read as a map of directions, it is worth asking what the image plane covers and what it does not, because the answer is short and it organises everything below.

The directions through a point in space form a projective plane. The image plane is a chart on it: each finite mark names one direction, and distinct marks name distinct directions. What the chart misses is exactly the directions lying in the plane through the eye parallel to the picture — their rays never meet the image plane, so they have no finite mark. Those directions form one line of the projective plane, which is the line at infinity of the picture and, when the ground is level, the horizon.

So a photograph is a chart covering all of the projective plane but a single line, and the two facts a reader has to hold about it are of quite different kinds. The missing line is a coverage gap: those directions have no mark, full stop, and a wider lens does not help because the deficiency is projective rather than optical. The two-halves ambiguity is not a coverage gap at all — those directions are covered, twice over, by one mark each.

That distinction is why the horizon behaves the way it does in every construction on this collection. A vanishing point is a covered direction and can be marked; the horizon is the boundary of what the chart reaches, and a construction that runs a line to the horizon and beyond has left the chart rather than found a point on it.

The same fact from the other direction: a mark that runs off one edge and returns from the other #

The reflection figure fixes the depth and moves the point. The complementary experiment fixes the line and slides one point along it, straight through the eye plane, watching the mark.

Two things are worth taking from that curve.

The first is that the two branches are not two curves. They are one curve on a line that closes, and the place they join is the vanishing point. A reader who follows the row of posts away from the camera sees the marks crowd toward the vanishing point; a reader who follows the same row backwards, past the eye, sees the marks crowd toward the same vanishing point from the other side. That is the closed projective line arrived at from the camera rather than from the algebra, and it is the more convincing of the two routes because nothing about it is a definition.

The second is the 66.58 pixels. The branches meet at infinity, so at any finite distance they are still apart, and the figure quotes how far apart at ±35 metres. That number is the honest statement of a limit rather than a claim that the two ends coincide at some particular distance.

The branch curve is one over the depth, which is the field’s other law #

The curve in the figure above is not an arbitrary shape and it is worth naming, because it is the same law that governs half of what this collection measures.

The mark’s position is $f x / z$ with $x$ fixed, so it is a constant over the depth — one branch of a rectangular hyperbola. The two branches of that hyperbola are exactly the two halves of the line, in front and behind, and the asymptote they share is the vanishing point. Nothing about the picture is discontinuous at the eye plane; what happens there is that a hyperbola passes through its own asymptote, which on a closed line is an ordinary crossing.

That $1/z$ is the field’s recurring law. Depth is a reciprocal measures it as a stereo disparity, where the useful quantity is a difference of reciprocals rather than of depths; where parallel lines meet is the same law at its limit, with the vanishing point as the value the reciprocal takes at zero.

Read against those, this essay is the observation that the law does not stop at zero. A reciprocal is defined on both sides of it, and the geometry that a camera implements is the geometry in which that is unremarkable — which is why a line is a closed curve is the companion piece to this one and why the two were measured with the same machinery. One approaches the closure from the algebra of the line and the other from the arithmetic of the lens, and they meet at the same mark.

What the sign was carrying, and who has to supply it #

If the mark does not record which side of the eye a point was on, then that fact is not in the picture, and anything needing it must get it from somewhere else. In reconstruction it has a name: cheirality, the handedness of depth, the requirement that a reconstructed point be in front of the camera that saw it.

The place this bites hardest is the four-fold ambiguity of a two-view reconstruction. Four cameras fit and one can see establishes that an essential matrix decomposes into four candidate poses. What is easy to assume, and false, is that the four differ in how well they explain the photographs.

The reprojection column is what makes this an argument rather than an assertion. If the four candidates fitted the marks to different accuracies, cheirality would be a tie-breaker among nearly-equal answers, and its role would be cosmetic. They fit identically — to 1.0 × 10⁻¹² pixels — so the choice among them is made by information the photographs do not contain.

Four is two twos, and this essay is one of them #

The four candidates are usually presented as a single fact about the essential matrix. They factor, and one of the two factors is precisely the bit this essay is about.

A two-view reconstruction recovers a rotation and a translation between the cameras. The essential matrix determines the translation only up to sign, because reversing the baseline changes the sign of the whole matrix, and an essential matrix is only defined up to scale. That is the same indifference to sign the single ray has, arriving one level up: the geometry knows the line the second camera sits on and not which way along it.

The other factor is a rotation. Given the baseline there are two rotations consistent with the same epipolar geometry, differing by a half-turn about that baseline — the classical twisted pair. A half-turn about the baseline sends everything in front of the second camera behind it, which is why it is invisible to the epipolar constraint and visible to a depth test.

Two signs, two rotations, four candidates. Both factors are the same species of ambiguity: each is a discrete choice the epipolar constraint cannot see, because the constraint is a statement about lines and both alternatives give the same lines. That is the shape of every exact ambiguity this collection records — a group acting on the answers that leaves the measurements alone — and it is the same shape as the two points a picture hides, where what is invisible is a pair of complex points rather than a sign. Neither is a numerical difficulty and neither improves with better marks — the figures here quote a residual of 1.0 × 10⁻¹² pixels across all four.

Reading it this way also says why the resolution is one test rather than two. Both factors are settled by asking which side of each camera the reconstructed points fall on, because both are, at bottom, a question about the sign of a depth.

The test is a control, and it stops working exactly where the scene stops obliging #

The usual demonstration of cheirality shows it selecting the right pose and stops there. That is the case where it works, and showing only that case would leave the reader with no sense of what the test is doing.

The right-hand end of that sweep is the interesting one and it is easy to misread. The test has not become wrong. It has stopped deciding. A scene straddling the eye plane admits no pose that puts all of it in front, so the criterion rejects everything and returns nothing, which is the honest behaviour for a criterion whose premise has failed.

That is worth setting beside the other refusals this collection records. A construction that returns a plausible answer where its assumption is broken is dangerous; one that refuses is merely limited. Cheirality refuses, and the sweep is what shows the refusal is a property of the scene rather than of the arithmetic — the epipolar residual sits at the arithmetic floor across the whole range.

Why a real camera does not have this problem, and what that costs #

A physical camera cannot photograph what is behind it. A lens has a barrel, a sensor has a back, and light from the rear does not arrive. So the ambiguity described here never produces a wrong photograph.

What it does produce is a wrong inference, and the distinction is the whole practical content of the essay. The camera resolves the sign by refusing to record the back half; the arithmetic does not, because the arithmetic is a map of directions and has no barrel. Anybody reasoning from marks rather than from photons is working with the map and inherits the ambiguity.

Two familiar consequences follow, and neither is usually presented as the same fact.

A plan recovered from a photograph can place a point behind the camera without the recovery noticing, because the algebra that places it is indifferent. Recovering the camera is exact on a scene that is entirely in front and says nothing about one that is not, and the cheirality test is the step that catches it afterwards rather than a property of the recovery.

And a vanishing point is the image of two opposite directions, not one. The mark that a row of posts converges on is the image of the direction the row runs and of the direction it runs backwards, which is why the same mark serves a row on either side of the camera and why nothing on the paper distinguishes them.

What the ambiguity is worth, which is not nothing #

An ambiguity that is nearly always resolved the same way looks like a technicality, and it is worth saying where it earns its keep.

The clearest case is a camera photographing a mirror. The virtual scene behind the glass is a genuine projection from a genuine centre — a mirror is a second camera measures exactly that — and the reflected camera sits behind the mirror plane, on the other side of a surface the real camera is in front of. A reconstruction treating the direct and reflected views as two cameras is working with a pair whose handedness differs, and the depth test is what notices.

The second case is a scene that legitimately straddles the eye plane, which is the right-hand end of the sweep above. A panoramic rig sees behind itself by construction, and for such a rig the phrase “in front of the camera” has no single meaning. The cheirality test is written for a pinhole with a forward half-space and does not transfer to an instrument without one; the sweep measures how it degrades, and it degrades by ceasing to select rather than by selecting wrongly.

The third is the honest one: most of the time, nothing at all. A scene wholly in front of both cameras resolves the four candidates immediately, and the sweep’s left-hand end says so — one candidate takes all the points and the other three take none. That is the ordinary case, and the reason the ambiguity is a footnote in practice rather than a problem.

What it means to say the picture “contains” it #

The title is deliberate and worth defending, because there is an obvious objection: nothing behind the camera is depicted, so in what sense is it contained?

In this sense. The image plane is a complete record of the pencil of rays through the eye. Every mark on it names a ray, and a ray extends both ways. So the picture determines, for every mark, a full line in space — not a half-line — and the half of that line behind the eye is as determined as the half in front. What the picture does not contain is the choice, one bit per mark, of which half a given piece of scene occupied.

That is a small quantity of missing information, and its smallness is the point. It is not faint or noisy; it is one binary fact, and no improvement in the photograph recovers it, because the quantity being measured is invariant under the operation the choice is about. The same shape of gap appears in a frame with a streak in it, where the density profile determines the path and not the direction of travel, and for the same reason: an integral, or a quotient, that is invariant under a reversal cannot report the reversal.

What this does not settle #

Two limits.

The first is that none of this is about lenses. A real optical system has a field of view, an entrance pupil with a size, and a housing, and every one of those breaks the pure direction-map picture in ways this collection measures elsewhere. The claim here is about the ideal pinhole, which is the model every reconstruction is written against, and the ambiguity is a property of that model rather than of any instrument.

The second is that the four-fold ambiguity is not the only thing cheirality is asked to do, and this essay measures only that. A reconstruction also has to decide the sign of its overall scale and, in some formulations, the orientation of the whole coordinate frame. Those are related and they are not the same question, and running them together would make the count of ambiguities look larger than it is.

One ray, two halves #

The object this essay is about is a line through the eye, and the finding is that a pinhole cannot tell its two halves apart because it was never looking at halves.

That is a loss of exactly one bit per mark, and it is the reason a two-view reconstruction arrives with four answers instead of one. The bit is cheap to supply when the scene is in front of the camera, which it almost always is; and the sweep shows what happens when it is not, which is that the criterion stops deciding rather than starting to lie.