Seeing the scene fences in the plane at infinity

Four cameras fit, and one of them can see used an assumption about the world to settle a question the algebra could not. An essential matrix decomposes into four camera pairs that all reproject every mark exactly, and requiring the courtyard to lie in front of both cameras chose one of them, 44 points to 0, 0 and 0. The assumption was not an equation. It was an inequality — depth is positive — and it settled a discrete choice completely.

That essay had the calibration. Without it, two pictures determine the scene only up to a projective transformation of space, and two views give shape and no size counted what that leaves free: fifteen numbers where a metric reconstruction has seven. Three of the eight extra numbers say which plane of the reconstruction is the plane at infinity — the plane whose points are infinitely far away, where parallel lines meet. A projective reconstruction does not know it. Parallel lines in it need not be parallel, because the plane where they should meet has not been identified.

The same inequality applies here, and it does something different. The ambiguity is not four answers; it is a three-parameter family of candidate planes. An inequality cannot pick one member of a continuous family. What it can do is fence part of the family off, and the fence turns out to have a precise shape.

Sending a plane to infinity #

The construction behind every figure here is short, and it is worth having exactly.

The true reconstruction of the courtyard is distorted by a fixed projective map of space, so that what is in hand is a projective reconstruction whose true plane at infinity is known to the figures and not to the recovery. Upgrading that reconstruction means choosing a plane and applying the map that sends it to infinity. If the chosen plane is the true one, the result is an affine reconstruction: parallels parallel, midpoints midpoints, only angles and relative lengths in different directions still wrong. If it is any other plane, the result is still projectively equivalent to the scene but places infinity somewhere it is not.

The slice fixes the third number of the plane at its true value and varies the first two. Every point of it is a candidate upgrade.

What a wrong plane does #

A plane is a boundary between two sides. Sending it to infinity sends everything on one side of it to the other side of infinity, which in an affine picture means the opposite side of the scene.

Choose a candidate a quarter of the way beyond the edge of the fenced region and three of the courtyard’s 44 points end up behind the cameras. They have not moved in any projective sense; the candidate plane passed between them and the rest of the courtyard, so the upgrade put infinity between them too. In the plan they vanish from the scene and reappear on the far side, and the blocks they belong to are torn in two.

That is the physical content of the inequality. A camera photographed the courtyard, so every point of the courtyard was in front of the camera, so no point of the courtyard can lie on the far side of infinity from the camera. Any candidate plane that would put one there is not the plane at infinity.

A plane inside the fence #

The region inside the fence is not the answer. It is the set of candidates the inequality cannot rule out, and a candidate well away from the truth can still be inside it.

Four fifths of the way from the true plane to the fence, every one of the 44 points is still in front of both cameras. The courtyard is visibly wrong: the blocks’ parallel edges converge, their rectangles are skewed into quadrilaterals, and distances across the scene are stretched unevenly. But no block is torn, and nothing lies on the wrong side of infinity. Every convex part of the scene is still convex and still in one piece.

That property has a name in the literature — a quasi-affine reconstruction — and it is a real intermediate stage between projective and affine. A projective reconstruction can split a single wall across infinity. A quasi-affine one cannot. It is wrong about parallelism and right about which points lie between which.

Why the fence is convex #

Each point and each camera contributes one condition, and the conditions have a simple form.

For a candidate plane, each point of the reconstruction lies on one side of it or the other, and so does each camera’s centre. The upgrade puts a point in front of a camera exactly when the point and the camera’s centre are on the same side of the candidate plane — with the side fixed by the true configuration. So the fence is the set of planes that have every point and every camera centre on the side the truth has them, and “on this side of a plane” is a linear condition on the plane’s numbers for each fixed point. The fence is an intersection of half-spaces, one for each point and each camera, and an intersection of half-spaces is convex.

The figure checks that rather than relying on it: the midpoint of pairs of points on the fence’s edge, taken across the whole outline, falls inside the fence every time. And the true plane is inside by construction, because the truth puts every point in front of every camera.

The same argument says which points matter. A plane that has every point of the courtyard on one side has the whole convex hull of the courtyard and the two camera centres on that side, because a half-space is convex. So a point in the interior of that hull adds no condition the outer points have not already imposed. The fence is set by the extreme points — the corners of the blocks at the edges of the scene and the two cameras — and a point in the middle of the courtyard, however precisely it is marked, rules out nothing.

How far the fence is #

Walking a candidate plane out from the truth along one direction shows where the fence stands and what crossing it does.

Along this direction every candidate up to 0.594 from the true plane keeps every point in front of both cameras. Past the edge the number of points behind a camera rises in steps: one at 0.600, two at 0.640, three at 0.680. Each step is the candidate crossing one more point of the courtyard. The edge is where the candidate plane first touches the scene’s hull.

The distance itself is in units of the plane’s numbers in the distorted reconstruction, so 0.594 is not a length anyone would recognise. What it measures is relative: the fence stands at more than half the width of a slice wide enough to contain every plausible candidate, which is to say the inequality leaves a great deal of freedom.

What each point buys #

The last figure measures that freedom as points are added.

With only the two cameras, 78.3 per cent of the slice is allowed. Requiring four points to be in front of both cameras brings it to 46.9 per cent. Eight points, 42.4; sixteen, 37.5; thirty-two, 37.3; all forty-four, 31.8.

The first few points do most of the work, and the step from sixteen to thirty-two does almost nothing. That is the hull argument showing itself. The points are taken in a fixed order through the courtyard’s list, and the sixteen already include most of the corners that define the scene’s outline; the next sixteen are mostly inside it. The last twelve include corners of the far block that the thirty-two had missed, which is why they move the share again.

And it never approaches zero. However many points are marked, the fence encloses a region, not a plane, because it is made of inequalities and an inequality removes candidates without ever pinning one down. Four cameras fit, and one of them can see needed only a region to be one of four points; here the region stays a region.

What the fence is for #

A region that holds the truth and rules out the tearing candidates is less than an answer and more than nothing, and it has two uses.

It is a safe place to search. Any method that recovers the plane at infinity from further information — vanishing points of lines known to be parallel, a known angle, a calibration — searches a three-parameter space, and a search that wanders outside the fence produces a torn reconstruction that looks absurd for reasons nothing in its residual reports. Restricting the search to the fence costs nothing and removes that failure. The scale of the saving is the count figure: two thirds of the slice, discarded for the price of an inequality.

It is the step the stratification quietly assumes. Two views give shape and no size set out the sequence from projective to affine to metric to Euclidean, each step bought by one piece of outside knowledge. The quasi-affine stage sits between the first two and is bought by a piece of knowledge so ordinary it is never listed: that the photographs are photographs of something the cameras could see. It does not upgrade the reconstruction to affine. It guarantees that whatever upgrade follows does not have to undo a torn scene first.

What the fence cannot do is distinguish the scene from a reflection of it. A narrow view keeps a second answer, inside out found a twin reconstruction whose depth order is reversed and whose points are all still in front of every camera. Cheirality admits both, here as there. The inequality is about which side of infinity a point is on, and a reflection about the middle of a scene moves no point across infinity.

What the two cameras rule out on their own #

The count figure’s first row is worth a paragraph, because it is the fence with no scene in it at all.

With no points required, only the two cameras’ centres must lie on the truth’s side of the candidate plane. That rules out every plane that passes between the true plane and either camera — a plane that would, sent to infinity, put a camera on the far side of infinity from the scene it photographed. In the slice that removes 21.7 per cent and leaves 78.3. Four points spread across the courtyard then remove another third of the slice at a stroke, because four points that are not coplanar already span a solid, and a plane that misses the solid of four spread points misses most of any solid the scene could make.

Everything after that is refinement, and it is refinement by the scene’s outline. The fence is a statement about the convex hull of the points and the cameras, and a hull is made by its corners. A scene with a few well-spread corners fences in the plane at infinity almost as tightly as a scene with thousands of points inside the same outline.

Where the three numbers come from instead #

The fence is not how the plane at infinity is usually found, and saying how it is found places the fence correctly.

Parallel lines meet on the plane at infinity. A family of parallel edges in the scene — the courtyard’s block edges running one way — has a vanishing point in each picture, and the two vanishing points are two images of one point on the plane at infinity. Triangulated from the projective cameras, they give that point. Three families in three directions give three points, and three points fix a plane. That route is an equation rather than an inequality, it is exact from exact marks, and it is the route recovering the camera takes in a single view when it reads a camera out of three vanishing points.

Its weakness is that it needs parallels the scene may not have. Foliage, rock, a crowd and a landscape offer no family of lines known to be parallel, and a scene with only two families gives two points on a plane that needs three. The fence needs nothing but the fact that the pictures were taken of something, so it is available in exactly the scenes that the vanishing-point route is not, and it gives less: a region instead of a plane.

The two also combine. A plane estimated from vanishing points read with error can land outside the fence, and when it does, the fence says so with no further information. A solver refining the plane from noisy parallels, like the adjustment where the adjustment stops describes, gains a constraint that costs nothing to evaluate and removes the one family of wrong answers that looks most absurd.

And once the plane is known, what is left is the part seven numbers no picture can name counted for a calibrated reconstruction, plus the calibration itself: five numbers of the cameras’ internal geometry that upgrade an affine reconstruction to a metric one, and then the seven no picture fixes. The fence sits at the very start of that sequence, before any of it.

The same line, from the other side #

The plane at infinity has been met from a single picture before. The picture contains what is behind the camera followed rays that leave a picture through the back of the eye and found them landing in the picture anyway, as the images of points behind the photographer. In a single view the line between in front and behind is the plane through the camera’s centre parallel to the picture, and a projective map of the picture does not know which side of it anything is on.

Two views put two such planes in the scene, one through each camera, and the fence is the region of candidate planes at infinity consistent with every point being in front of both. It is the same distinction — in front of the eye or behind it — made twice and intersected, and it is exactly as strong as that sounds: strong enough to forbid tearing, and too weak to identify infinity.

What this does not settle #

One slice. The plane has three numbers and the figures vary two, holding the third at its true value. The fence is a convex solid in three dimensions and the slice is one cross-section of it; its share of a three-dimensional box was not computed.

One distortion. The projective map applied to the true courtyard is fixed. A different map places the same fence differently in plane-space; the share of a slice depends on the map and on the slice’s width, and the numbers here belong to both. What does not depend on them is the fence’s convexity, the true plane’s being inside it, and the hull argument for which points bound it.

Exact marks. Every point here is reconstructed exactly. With reading error, points near a candidate plane can be placed on the wrong side of it, and the fence’s edge becomes a band. Its width was not measured.

Still open: how a third camera tightens the fence #

Every camera centre is one more point that must lie on the truth’s side of the candidate plane, and every picture from a new direction sees the scene’s hull from a side the first two did not. The count figure found that more points in the same scene buy less and less, because interior points add nothing. More cameras are not interior points.

The question that leaves is how fast the fence closes as views are added around the courtyard: whether a third camera on the far side of the scene removes most of what is left, whether the fence ever becomes small enough to locate the plane at infinity to a useful precision without any other information, and how its size compares, view for view, with what the same cameras would give if their focal lengths were simply known.