A plane's coefficient reaches as far as its parallax

A parallax length is a height over a depth took a courtyard photographed twice, fitted the ground’s map from one picture into the other, and showed that each raised mark’s place in the second picture is the map of its place in the first plus a multiple of the epipole. That multiple — the coefficient — is the mark’s height above the ground over its depth from the first camera, times one number for the pair of pictures. On the courtyard’s twenty-two raised marks it held to $4.5\times10^{-14}$, read the same from three different second pictures, and survived half a pixel of reading error at about one per cent.

Every one of those marks was a corner of a block standing on the courtyard’s own floor, inside the patch whose ground marks fixed the map. A real photograph has one plane that is known and a great many points that are nowhere near its marks: the top of a tree across the road, a window three storeys up, the bottom of a pit, a car forty metres down the street. The essay closed by asking whether the coefficient reaches them. A homography fitted to a small patch and extrapolated usually does badly, and if the coefficient inherits that, the relief map it draws is only as wide as the floor that was measured.

The answer below is that the floor’s width is not the thing that runs out, and that the thing which does is visible in the picture.

The algebra has no edge #

Before any reading error, the question is whether the coefficient means the same thing everywhere. Its derivation said it should: a point’s ray from the first eye crosses the plane somewhere, the map sends that crossing to its place in the second picture, and the point itself lies along the line from there to the epipole by an amount set by its height over depth. Nothing in that argument asks where the point is.

The measurement agrees. Points three metres below the ground, twelve metres above it, seventy metres past the courtyard, and exactly at the first camera’s height — where the ray from the first eye runs parallel to the ground and meets it only at the horizon — all return their scaled height over depth to between $10^{-16}$ and $10^{-14}$. Points above the eye, whose rays meet the ground behind the camera, return it just as well: the map is a matrix acting on rays, and it does not care which side of the eye a ray crosses the plane on. Points below the plane get negative coefficients, as a height below the plane should.

So exactly read, the coefficient reaches everywhere either picture sees. What remains is the only question that matters for a photograph: how its error grows when every mark is read the way a camera reads it.

A plan of the error #

The hero figure puts probe points one metre above the ground on a grid, keeps every one that both pictures see, and reads each with every mark in error by a pixel — the ground marks, the raised marks that give the epipole and the scale, and the probe’s own two marks — over sixty trials.

Beside the courtyard the error is one to two per cent. Thirty metres out it is eight to nineteen. It grows steadily with distance and the plan looks exactly like the extrapolation failure the question expected — a number that is good near the floor and poor away from it. The next three figures show that the plan is misleading about why, and the reason matters because it changes what a photographer can do about it.

Height helps, and the eye’s height is nothing special #

The plan holds every probe at one metre. Letting the height vary at a fixed position separates two things the plan runs together.

At every position, the error is worst just off the plane and falls steeply as the point rises or sinks. Twelve metres out, a point 0.6 m up is read to 6.7 per cent and a point twelve metres up to 0.68 — ten times better, at the same distance from the marks and therefore with the same extrapolation of the map. Twenty-eight metres out the gain is fifteen-fold. Below the plane the curves are nearly mirror images: a point two metres into a pit is read about as well as one two metres up a wall.

Among the marks themselves the tallest probes turn slightly worse again, because a point twelve metres above the courtyard and a few metres from the camera leaves the upper edge of the pictures’ shared view and its two marks sit far from any of the marks that fixed the geometry. That is a detail. The general pattern is that height is on the coefficient’s side, and so the question “how far does it reach” has no answer in metres along the floor alone: it depends on how far off the floor the point is, for its distance.

The dashed line in the figure is the first camera’s height, 1.6 m. Nothing happens there. The ray from the eye to a point at that height is parallel to the ground, so the “crossing” the map acts on is a point at infinity, and it is natural to expect trouble. There is none: the map sends points at infinity on the ground to the horizon in the second picture as reliably as it sends near ones, and a point just above the eye, whose ray crosses the ground behind the camera, is read as well as one just below. The algebra was homogeneous from the start, and homogeneous coordinates were invented precisely so that a crossing at infinity is not a special case. A point is a line over there makes the same argument for a picture’s own horizon.

The map does not blow up #

The expected mechanism of failure was the map itself: fitted from marks that span a few metres, it would be wrong by more and more as it was carried further from them. Measured directly, it is not.

Fitted to the courtyard’s twenty-two ground marks, the map misplaces a point of the ground by 0.65 px among the marks, 2.35 px ten metres out, 3.41 px at thirty and 4.23 px at 130. The error grows at first and then flattens. It does not grow in proportion to distance, and it never will, because distance on the ground is the wrong measure of how far the map is being extrapolated.

The map acts on the picture, and in the picture the distant ground crowds toward the horizon. Thirty metres out the ground is 33 rows below the horizon in the first picture; at 130 m it is under nine. A map carried from the courtyard, which fills the lower part of the frame, to the horizon, has been extrapolated over a bounded stretch of picture — a couple of hundred pixels, not a hundred metres — and its error at the horizon is a finite number set by how well the marks fix the horizon’s image. The horizon at eye level is the same fact seen from the drawing side: every point of the infinite ground fits inside a band below one line.

A wider floor of marks helps, and the figure measures by how much: five by five marks spread over twelve metres cut the far error to 1.14 px, and over twenty-four metres to 0.85. But even the courtyard’s own marks, which span barely five metres, leave the map a few pixels wrong at infinity rather than wrong without limit.

Where the error actually comes from #

If the map’s error flattens, something else must be growing to produce the plan’s eight to nineteen per cent. The coefficient’s error has three sources, and reading one set of marks with error at a time separates them.

The raised marks that give the epipole and the overall scale contribute least: under half a per cent at every distance, barely changing, because twenty-two parallax lines fix the epipole well and their average fixes the scale better still. The other two grow. The probe’s own two marks cost 1.2 per cent among the courtyard’s marks and 6.5 per cent at thirty metres; the ground marks, through the map, cost 0.2 per cent among them and 8.0 per cent at thirty. The three add in quadrature to the whole, which is what independent errors should do.

Both growing shares have the same denominator. A point’s coefficient is read from how far its mark in the second picture sits from where the map sent it — its parallax — and an error of a pixel, whether in the point’s own mark or in where the map sends it, is a fraction of that parallax. A point one metre up and thirty metres out has a small height over depth, so its parallax is a few tens of pixels; a pixel is a large fraction of that. The point’s own share grows purely because the parallax shrinks. The map’s share grows because the parallax shrinks and because the map’s pixel error rises from 0.65 to 3.4 px before it flattens, which is why, past fifteen metres or so, the map’s share is the larger of the two with the courtyard’s marks.

One curve, in pixels #

That analysis says the coefficient’s error should be governed by one quantity in the picture: the point’s parallax after the map. It can be tested by putting every probe of every sweep on one plot.

A hundred and two probe points — below the plane and above it, beside the courtyard and forty-six metres past it, with the second camera beside the first and stepped toward the scene — fall on one line. The error is about 260 per cent divided by the parallax in pixels, to within a factor of 1.35. The slope on the logarithmic axes is −0.86 rather than exactly −1 because the map’s share carries its own mild growth with distance, and distant points are also the short-parallax points; on the courtyard’s marks, that is the whole of the map’s influence.

This is the answer the question was reaching for, in a form a photographer can use. The coefficient does not have a reach in metres. It has a reach in pixels: a point is read to 1 per cent if its parallax is about 260 px, to 10 per cent if it is 26, and it does not matter whether that parallax belongs to a low wall nearby, a rooftop across the street, or the rim of a pit. The number can be read off the photographs before anything is computed, because the parallax is visible as soon as the map is applied: it is the distance between where the map sends a point and where the point actually is.

How flat is flat enough found the same kind of law for recovering the cameras at all: the pose was useless until the out-of-plane parallax reached about ten times the reading error. The coefficient is the pointwise version. Every raised point carries its own out-of-plane parallax, and its coefficient is exactly as good as that parallax is long.

What a wider floor buys #

The map’s share is the one part a photographer can reduce by marking more of the plane. The last figure measures how much that is worth.

Twenty-five ground marks spread over a floor twenty-four metres square, against the courtyard’s twenty-two over five metres, divide the map’s share by 4.5 forty-five metres out — from 12.1 to 2.7 per cent. The whole error falls only by 1.7, from 14.3 to 8.4 per cent. The rest is the point’s own reading of its own short parallax, which no amount of floor can lengthen.

That sets a practical rule. Marking a wide floor is worth doing when the points of interest are far away, because it removes the part of the error that belongs to the map. It is not a way of reaching points with little parallax: a car forty-five metres down the road, one metre tall, is read to no better than eight per cent from a single pair of pictures whatever the floor, because its displacement off the plane is a handful of pixels. What reaches such a point is a longer baseline — a second picture taken further from the first, so the same point shows more parallax — and that is a choice about the cameras, not about the plane.

What the relief map is, then #

Two marks off a known plane find the other eye established that raised points’ displacements all point at the epipole; the parallax length turned each displacement’s length into a height over depth. The question here was whether that relief map is as wide as the floor. It is wider, and its edge is somewhere else.

The relief map is exact everywhere the pictures see, and its precision at any point is set by one thing visible in the pictures: how far the point’s mark sits from where the plane’s map sends it. Points high above the floor or deep below it carry long parallax and are read well at any distance, including far above the eye and far past the marks. Points just off the floor and far away carry short parallax and are read poorly, and the floor’s marks contribute a share of that which a wider floor can reduce but not remove. A single floor is enough to rank everything in a room that stands well off it; it is not enough to rank a kerb against a paving slab forty metres down the street, and nothing about the floor could make it so.

This also changes what “extrapolating a homography” means for a plane seen in perspective. The usual warning — a map fitted to a small patch is unreliable away from it — is true on the plane’s own coordinates and misleading in the picture’s. The picture compresses the whole infinite plane into a band below the horizon, and a map fitted to the bottom of that band is extrapolated over a bounded stretch of pixels, not an unbounded stretch of ground. A plane’s map is fixed by four marks, and any four that span the band’s lower part already constrain where its horizon goes.

What the sweep leaves out #

The marks are read with independent, round errors. A real matcher’s errors are correlated between neighbours and larger on low-contrast texture, and the far ground is exactly where texture is finest and matching worst. The figures price a pixel; a real photograph’s far pixels are worth less.

The pictures are perfect pinholes. A lens’s distortion bends the far ground most, near the edges, and a map extrapolated to a horizon it cannot see straight will carry the distortion into every far point’s parallax. Fitting a lens from straightness alone is the prior step such a photograph needs.

One second picture at a time. Every figure reads the first picture against one second one. The coefficient is independent of the second picture, so several of them could each supply a reading of the same point and be averaged; for a point with short parallax in each, that is the other way of buying precision a floor cannot.

Still open: what several second pictures buy a point with short parallax #

The coefficient belongs to the point and the first camera, and the parallax length found three second pictures reproducing it to $4\times10^{-14}$ when read exactly. Read to a pixel, each second picture gives an independent estimate whose error is about 260 per cent over that picture’s parallax, and those parallaxes differ: a second picture stepped far to the side shows a far point more parallax than one stepped forward.

That makes the combination a weighting problem with a definite answer in the pictures. Each estimate should count in proportion to the square of its parallax, and the combined error should then be set by the root sum of squares of the parallaxes rather than by any one of them. The measurement that settles whether that is the whole story reads a point with short parallax — a metre-high kerb forty metres out — from the first picture against four second pictures at different baselines, combines the four coefficients weighted by their parallaxes, and asks whether the combination reaches the error the summed parallax predicts, or whether the map’s share, which the four readings share through the first picture, sets a floor that no number of second pictures goes below.