Second pictures reach a floor unless they go round

A plane coefficient reaches as far as its parallax found how far a known plane’s measurement of height extends. Exactly read, the coefficient on the epipole gives a point’s scaled height over its depth anywhere both pictures see. Read to a pixel, its error is one over the parallax at that point, and it grows as a point recedes until a kerb forty metres out is effectively unmeasured. What sets the reach is visible in the picture: how many pixels of parallax a point has.

So a point with short parallax wants more pictures. That essay closed on this, and the parallax length had already found that three second pictures, read exactly, reproduce one coefficient to $4\times10^{-14}$. Each second picture is an independent estimate of the same number, with an error set by its own parallax. The arithmetic of combining them looks routine. Weight each by the square of its parallax, and the combined error should come from the root-sum-square of all the parallaxes. It should be as if the pictures had pooled their parallax.

There is one thing wrong with that picture, which the earlier essay flagged and could not measure. Every second picture is compared against the same first picture. So what the first picture got wrong, all of them get wrong in the same way.

The arrangement #

The point is a kerb a metre high, 40 m from a camera 1.6 m above the ground, with a focal length of 900 px. A second camera moved sideways by $b$ sees it with parallax $p = fbH/(Zh)$, so 14 px for each metre of baseline. Take four second pictures, at baselines of 0.3, 0.6, 1.2 and 2.4 m. Their parallaxes are 4.2, 8.4, 16.9 and 33.8 px.

Each has a reading error of 1 px in the first picture, which is shared, and 1 px in its own second picture. There is also the error in the plane’s own map at 40 m. That is well past the marks the map was fitted from, so an error of 2 px at that distance is a reasonable working value, and it is independent from one picture to the next. The plane’s map is otherwise taken as read: near a flat scene the map from the first picture to any second is close to a translation, so what the first reading gets wrong lands on every second picture as the same error.

Weights count for more than pictures #

Consider three ways of combining these.

The shortest baseline alone has an error of 58.1 per cent, and the longest alone 7.3 per cent. So far, this just restates the earlier reach. Averaging the four estimates with equal weights gives 18.9 per cent, which is much worse than using only the longest: the three short baselines have imported their own much larger errors. A person who assumes that more pictures are automatically better, and who averages them, has here reduced the value of the fourth.

Weighting each by the square of its parallax, as least squares says to, gives 7.1 per cent. That is hardly better than the longest baseline on its own, at 7.3 per cent. The arithmetic of pooled parallax predicted more than that. Summing the squares of the parallaxes gives 39.1 px, and the second-picture error of about 2.2 px over that would be 5.7 per cent, before the first picture has been accounted for. The rest of the difference is the shared reading.

The check on this is simple and direct. Simulated with independent errors in 4,000 trials, the equal-weight and weighted results both match the closed forms to within five per cent.

The first picture is shared #

Split the weighted result’s error into its parts.

With no reading error at all and 2 px of map error, the coefficient comes to 5.1 per cent, from the second pictures’ independent errors alone. With 1 px of reading and no map error, it comes to 4.9 per cent, of which 4.2 per cent is the shared first reading. The two combine to 7.1 per cent. So the first picture, with its single pixel of reading error, is responsible for most of what is left, even with every picture weighted correctly.

The reason is geometric. A first picture read wrongly by $\delta$ moves the plane’s prediction of where the point should be, in every second picture, by about $\delta$. The parallax in each then looks larger or smaller by that amount. If all the parallaxes run in one direction, they all look larger or smaller at the same time. The error in each picture’s coefficient is $\delta/p_i$, all with the same sign, and they do not cancel.

Adding second pictures, then, runs into a floor.

Take a baseline of 2.4 m and make as many second pictures there as desired, all in one direction. The error is 7.26 per cent with one, 4.44 with four, 3.39 with sixteen, 3.08 with sixty-four, 2.99 with two hundred and fifty-six. It approaches 2.96 per cent and never reaches it. That number is simply one pixel over 33.75 px of parallax, or the first picture’s reading error, and it has the closed form that the earlier essay’s reach would have predicted for one perfectly read second picture.

On one line, an unlimited number of second pictures is worth about one perfect second picture. That is the answer to the question the earlier essay raised about the map’s share: whether the combination reaches the error the summed parallax predicts, or whether something shared sets a floor. A floor, it is, and what sets it is the first picture rather than the map. The map’s error is independent in each picture and averages away. A further consequence is worth stating. Spending more second pictures at a shorter baseline is capped by a higher floor, so it never catches up.

Two short pictures, at 0.3 and 0.6 m, give 27.5 per cent. Two long, at 1.2 and 2.4 m, give 6.9 per cent. One at 2.4 m gives 7.3 per cent, and one longer, at 4.8 m, 3.6 per cent. Sixteen at 0.6 m give 13.6 per cent. The ordering is clear enough, and there is a clean rule of thumb: first find the longest baseline available, since it sets the floor at one over its parallax. A small number of pictures at that baseline reach near that floor, and any more add little.

How many are worth taking #

The floor makes it possible to say in a line how many pictures on one line are worth the trouble. With a reading error $\sigma_a$ in the first picture and $\sigma_b$ in each second picture, $n$ pictures at one baseline have a variance of $(\sigma_a^2 + \sigma_b^2/n)/p^2$. The first term is the floor and the second is what averaging removes. To come within 10 per cent of the floor, the second term must fall to 21 per cent of the first, so

$$n = \frac{\sigma_b^2}{0.21\,\sigma_a^2}.$$

With a read of a pixel on every picture and two more pixels of map error on each second picture, that is $\sigma_b^2 = 5$, and $n$ is about 24. With a poorer map at six pixels, $\sigma_b^2$ is 37 and $n$ is about 176, since now there is a lot more independent noise to average. If a map is excellent and the second reading is the only error, $\sigma_b^2 = 1$ and $n$ is 5. So the pictures needed at a stated baseline depend on how much of the error is private to each one, and the rule is plain: keep adding until the private part is small beside the shared part, and then stop, since there is nothing left to buy. A flight planner working from this does not need a simulation. A person who has read the map’s error from its fit, and who knows what a pixel of reading means in their matcher, has what they need.

That gives one further reading of the earlier result on the map’s share. How flat is flat enough asked how close to a plane a scene has to be before the map is good enough for a stated purpose, and a map’s error at a far point can be read from that. So the two limits meet here. A long way past the ground marks, the map’s error is large, and many second pictures are worth having, since they are what averages it down. Near the marks, the map is excellent, and few pictures are worth having, since the first picture’s own reading dominates almost at once.

Going round #

The floor applies because all the parallaxes run the same way, and a reading error in that direction looks exactly like a change in the coefficient. Now move second pictures to different sides of the first, so that their parallaxes point in different directions. A reading error $\delta$ of the first picture is one vector, and it affects each second picture by its component along that picture’s parallax direction. Parallaxes running at different angles now disagree about what the coefficient would have to be, if the first reading were correct. A fit treating both the first reading and the coefficient as unknowns has enough information to work out where the first reading really was.

In the same figure above, sixteen second pictures spread evenly round the point reach 1.66 per cent, sixty-four reach 0.83, and two hundred and fifty-six reach 0.41. That is falling as the square root of their number, with no floor. Going from one line to a ring has changed the kind of thing the combination is.

A reader may be tempted to think that a little spread would be enough, and it is not quite.

With sixteen second pictures at 2.4 m: in one direction, 3.39 per cent. Over 30°, 3.26 per cent. Over 90°, 2.62 per cent. Over 180°, 1.94 per cent. Spreading the pictures over a right angle takes off about a quarter of the floor’s error. The reason is that a fit needs a component across the first direction, and a small spread provides very little of it, in proportion to the sine of the spread. A bundle’s twist is tested by its own error finds a similar need for variety in another setting, where a fit cannot check something its own measurements all share. That result lines up with the way a practical survey is usually laid out, as two marks off a known plane suggests in another setting: a few pictures from well-separated sides are worth more than a long row of them.

Seven numbers no picture can name is the formal version of the point that follows. What is happening here is that a first picture read in isolation has an error nobody can check. Pictures from several sides make it checkable, and they turn the first picture from an authority into one more measurement. So it is not strictly a first picture any more. The asymmetry between “first” and “second” came from the plane-and-parallax way of stating the problem, and a full bundle adjustment has no such asymmetry: every picture is a measurement of the point, and the point’s position is a free unknown. That is the sense in which the fit above is a small bundle adjustment, and it is why a bundle’s error behaves in the same way.

Reading the same thing from a practical angle #

Where does a floor matter in practice? It arises wherever someone reads a small height against a known plane, from a fixed reference picture, using a line of later pictures. A camera on a vehicle moving along a road, comparing later frames with one reference frame, is a clear example. So is a drone flying a strip over a nearly flat site. Their parallaxes are all in one direction, set by the direction of travel, and however many frames they compare, a reading error in the reference frame is frozen into the result.

Two remedies follow, in increasing order of cost. Average the reference frame’s own reading by using several reference frames, which amounts to doing the pool in both directions and is a good deal better. Or use the frames as symmetric measurements in a joint fit, which removes the idea of a reference altogether. Both are standard practice in the fields concerned, and the numbers here give their value. A joint fit along a line only recovers what the first picture’s reading can give: it does not remove the floor, because a line does not contain the needed direction. So for a vehicle driving in a straight line, the first frame’s pixel matters no matter how the fit is done. A second pass along a different line is what would help.

A flat scene fixes no second eye reached its own limit by a different route, a limit in the plane’s own family of solutions. The one here is simpler, and it would hold even with a plane and an epipole known exactly. It does not concern the pictures’ geometry. It comes from how many independent readings there really are of the thing being measured, and a person counting pictures may easily count more than there are.

A fitted map shares its error, so the real floor sits higher #

A known plane and known epipoles. One flight, two pictures has the same frozen reference in a survey’s own terms. The map’s error is modelled as an independent term per picture, of a stated size. In reality the map is fitted from marks, and each picture’s map shares some of its error with the others, through the first picture’s reading of those same marks. That would raise the floor somewhat, and it is not measured here.

The map as a translation. Near a flat scene, for small baselines, this is close to true, and it is what makes the arithmetic linear. For a very long baseline or a strongly tilted plane, the first picture’s error lands on each second picture with a different size and direction, which would itself provide some of the variety that a ring provides.

Independent readings of 1 px. A matcher’s errors are often correlated from one picture to the next, since it uses the same patch from the first picture. That makes the shared part worse than the model says.

All second pictures have the same reading quality. With real ones, images from further away are lower resolution, and a kerb 40 m out is a small target. How much of this holds for a real low-contrast edge would need a real matcher’s behaviour in the model.

Still open: what a plane fitted from the same pictures shares #

In the arrangement here, the plane is known and its error is independent per picture. A real plane is fitted from ground marks read in the same pictures, by the same first reading. So each second picture’s map has an error that shares a component across all of them, as the first picture’s reading of the ground marks is the same in every fit. That common error is a shared tilt of the plane, and a shared tilt is a coefficient that varies across the picture rather than a single offset.

The measurement that settles what it costs fits the plane from stated ground marks in the first picture and each second picture, reads a point at stated distances beyond the marks, and asks whether the first picture’s reading of the ground marks adds a second floor to the one measured here; how far past the marks it starts to dominate; and whether going round the point removes that floor as well, or only the one set by the point’s own reading.

More pictures only help with the errors they do not share. A first picture read once is read once, however many second pictures compare against it, until something in the arrangement lets it be checked.