A fit weighted by the miss trusts only the surface

The residual does not warn fitted a straight line to the stick a stereo pair puts back when it looks into water, and found the fit’s residual running the wrong way. It was 0.535 mm at a level baseline, where the two rays to every point meet exactly, and fell to 0.038 mm at sixty degrees of roll, inside the band where the rays miss by more than a pixel can explain. Meanwhile the fitted line was 285 to 584 mm short of the stick’s metre. The one quantity that behaved was the rays’ own miss — zero at the two rolls whose reconstructions are real images, positive between them — and the essay closed by proposing to put it to work: weight each point of the fit by the inverse of its own rays’ miss, so that the fit trusts the points the rays agree about.

It is the obvious repair, and it is the kind of repair a careful pipeline actually makes. A third ray is worth what its picture is worth is the case for weighting an observation by what it is worth, and the miss looks like exactly the number that says what each reconstructed point is worth. The essay asked three things of the weighted fit: whether its direction comes closer to the stick’s, whether its residual falls where the miss falls, and what becomes of it at the two exact rolls, where every weight is infinite.

The answer to all three comes from one fact about where the miss is zero. It is worth establishing that fact first.

The miss is zero at the surface, at every roll #

The miss is not scattered along the stick the way a matcher’s errors would be. It is a smooth function of the geometry, and it has a zero in a place that no roll of the baseline can move.

The point where the stick enters the water sends each eye a ray that crosses no surface on the way, so it bends nowhere and the two rays to it meet exactly. That is true at a level baseline, at an upright one, and at every roll between. Below the surface the rays bend, and bend differently for the two eyes, and the miss grows with the depth of the point. At forty-five degrees of roll it runs from nothing to 3.13 mm at the tip, 3.86 mm per metre of depth on average and within 0.21 mm of strict proportion; at fifteen and seventy-five degrees it grows half as fast.

So a weight that is the inverse of the miss is not a weight about individual points at all. It is a weight about depth — trust the shallow end, distrust the deep end — and it says the same thing at every roll. The curves differ in how steeply they climb, not in where they start.

The zero at the surface is not a feature of this stick or this pool. A point on a refracting boundary is the one point refraction cannot misplace, because a ray from it reaches the eye without crossing the boundary at all; it is reconstructed exactly by any pair of eyes, from any roll, at any index. Every object that breaks the surface carries such a point at its waterline, and every one of those points is, to a triangulation, the most trustworthy point in the scene. It is also the only point on the object whose position was never in doubt. The measurement that most wants a weight — how deep does this go, and at what angle — is the one the weight steers away from.

That is already enough to predict what the weighted fit will do, and the prediction is not the one the proposal hoped for.

Unfloored, one point takes all the trust #

The inverse of zero is infinite, which is why the proposal itself said the weighting would need regularising. The plainest regularisation adds a floor in quadrature: weight each point by one over the miss squared plus a floor squared. A floor of a billionth of a metre is the literal proposal with the division by zero removed.

With no real floor, the entry point takes all but two ten-billionths of the fit’s weight. The next point down, a couple of centimetres into the water, has a miss of 0.09 mm and gets a hundred-and-forty-billionth of the whole. The fit is a line pinned to the one point where the stick enters the water, with its direction settled by points it trusts ten orders of magnitude less.

A floor of a tenth of a millimetre — small against the miss, which reaches three — spreads the weight, but the entry point still takes 42 per cent on its own. Only at a floor of the size of the reading error, the 2.85 mm a pixel covers at this range, do the shares come out near even: 1.27 to 2.80 per cent each, against the 2.04 an unweighted fit gives every point. A floor that large is below most of the miss, so the weights barely differ, and the weighted fit is close to no weighting at all.

There is no floor between these that is not arbitrary. Below the reading error the weighting prefers points for agreeing more closely than the pictures can measure; at the reading error it has nothing to prefer them for.

The weighted residual runs the wrong way harder #

With the weights understood, the residual follows.

Weighted with no floor, the residual is zero — under a hundred-thousandth of a millimetre — at every roll from ten degrees to seventy-five. A line pinned to one point and weighted almost entirely at that point leaves almost nothing unexplained, because almost nothing is being asked of it. At the two exact rolls, level and upright, the rays meet at every point, every miss is zero, every weight is the same, and the fit is exactly the unweighted one: 0.535 and 0.371 mm.

So the weighted residual is largest precisely at the two arrangements where the reconstruction is an honest image of something, and zero at every arrangement where it is not. That is the failure the residual does not warn measured, made total. The unweighted residual was merely smaller where the trouble was; the weighted one reports perfection there.

Floored at the reading error, the weighted residual lies on the unweighted curve to within 0.003 mm at every roll. Nothing is repaired and nothing is made worse; the weighting has simply been turned down until it does nothing. And the rays’ miss, root-mean-square over the stick, is still the only curve in the figure that rises where the rays disagree — 1.94 mm at forty-five degrees.

The floor decides the number #

The two extremes are two answers, and between them is a dial.

At forty-five degrees of roll the weighted residual climbs smoothly from nothing, at floors of a ten-thousandth of a millimetre, to the unweighted 0.214 mm at floors of a pixel and above. It passes half that value at a floor of 0.30 mm, which sits inside the range the miss runs through along the stick. The curve is smooth and monotone and has no feature at any floor that could be read as the right one.

That is the finding the proposal did not expect. A weighted residual is a number the analyst sets by choosing the floor, and at any floor it is a statement about how much the analyst chose to trust the surface. The one floor with a reason behind it — the reading error, below which a smaller miss is not evidence of anything — returns the residual that did not warn in the first place.

Weighting does not straighten the line either #

The first thing the proposal asked was whether the weighted fit’s direction comes closer to the stick’s. Even a residual that warns badly would be worth something if the fit it came from were better.

It does not, by any amount that matters. With no floor the weighted line is up to 0.26 degrees worse than the unweighted one at the shallow rolls and up to 0.22 degrees better at the steep ones; with the floor at a pixel the two are within 0.022 degrees everywhere. The unweighted error over the same sweep runs from 10.4 to 14.3 degrees.

The reason is visible in the first figure. The points the weighting trusts are the shallow ones, and the shallow points are bent too — less than the deep ones, but in the same direction, because a straight stick in water is a kink and a curve: the reconstruction leaves the surface already 14.96 degrees off the stick at a level baseline, before any depth has accumulated. Choosing which bent points to trust chooses between bents. It cannot find the straight stick, because no point of the reconstruction is on it except the one at the surface, and one point has no direction.

The miss measures agreement, not accuracy #

That leaves the miss itself, which every figure so far has treated as the honest number. It is honest. The question is what it is honest about.

Plot each roll as one point, the miss across and the shortfall up, and the two rolls where the rays meet exactly are the two extremes of the error. A level baseline returns the stick 285 mm short at a miss of zero; an upright one returns it 584 mm short at a miss of zero. Between them the miss rises to 1.94 mm and falls again while the shortfall climbs steadily from one end of the range to the other. There is no relation between the two quantities to exploit, because they are measuring different things.

The miss measures whether two eyes agree. At a level baseline they agree perfectly, and what they agree on is the sagittal image of the stick — a real image, formed by rays that genuinely meet, and a real image of the wrong thing. A point under water has two depths is about why there are two: the rays from one point pass through two focal lines, and a pair of eyes side by side reads one while a pair one above the other reads the other. Both are consistent. Both are wrong.

So the miss can say that a reconstruction is a compromise between two images; it cannot say that either image is right. At the two rolls where it is zero, nothing available to the pair — not the miss, not the residual, not any weighting of one by the other — distinguishes the reconstruction from a real stick of a different length and lean. What would is information from outside the two pictures: the index of the water, a third view from a different height, or the known length of something in the scene.

What a pipeline can do with the miss #

None of that makes the miss useless. It changes what it is good for.

It is a flag, not a weight. A miss larger than the reading error says, point by point, that the two rays do not come from one place, and that the reconstruction there is a compromise nothing in the scene supports. That is worth reporting beside every reconstructed point, and the midpoint is a choice of ruler already argued that a midpoint which discards it has thrown away the evidence that the question was ill-posed. What the miss cannot be is a weight that repairs the fit, because the points with the smallest miss are not the most accurate ones; they are the shallowest.

Its zero is not a certificate. A reconstruction whose rays meet everywhere is consistent, and a consistent reconstruction through water is a real image — the sagittal stick or the tangential one. A pipeline that reports “rays meet, residual small” for a level stereo rig looking into a pool is reporting the truth about the rays and nothing about the pool.

And the fix for refraction is refraction. Every number in these essays is the water’s; the residual does not warn ran the identical reconstruction in air and got the stick back to $4\times10^{-13}$ m. A triangulation that traces each ray through the surface with the water’s index has no miss to weight and no compromise to fit, and the reconstruction is exact. The repair is not a better statistic applied after the geometry is wrong. It is the right geometry.

Where this reading stops #

The surface is flat and known. Every ray here crosses one level surface of water whose height is given. A wavy surface has no fixed entry point and a miss that is not zero anywhere, and the pinning found here would not happen in that form — though the weights would still sort points by how the surface happened to bend their rays, which is not accuracy either.

The floor is added in quadrature. Other regularisations exist — a cap on the weight, a weight that falls off with the miss rather than dividing by it — and each chooses a different curve through the dial in the fourth figure. None was measured, and nothing about their construction suggests any would find the straight stick when every trusted point is bent.

The stick is in the plane through the eye. As in the two essays before this one, the stick leans directly away from the eye, which is what gives the two exact rolls their symmetry. A stick turned out of that plane has no exact roll and no zero miss except at the surface, so the extremes of the missvserror figure would not exist; the pinning at the surface would.

And one stick. A fit to many sticks at once — a floor, a grid, a scene — would carry many entry points, one per object, each with a zero miss. A weighting by the miss would then pin the fit to the waterline of every object in view, which is a picture of the surface rather than of anything below it.

The weighted fit, weighed #

The miss between two rays to a submerged point is zero at the surface at every roll of the baseline and grows with depth, so weighting a fit by its inverse is weighting by depth. With no floor the entry point takes all but two ten-billionths of the trust and the weighted residual is zero at every rolled baseline, returning to the unweighted 0.535 and 0.371 mm only where the rays meet exactly — the failure the residual already had, made total. With a floor at the reading error the weighted fit is the unweighted one to 0.003 mm, and between the two the reported residual is whatever the chosen floor makes it.

The weighted direction is within a quarter of a degree of the unweighted one against an error of ten to fourteen degrees, because every trusted point is bent the same way as the distrusted ones. And the miss itself is zero at a level baseline where the stick is 285 mm short and at an upright one where it is 584 mm short. It measures whether two eyes agree, and two eyes agree perfectly about two different wrong sticks.

Still open: whether a third eye at another height breaks the agreement #

The two exact rolls are exact because each pair of eyes lies in one of the two families of planes the refraction is symmetric about. A level pair reads the sagittal image and an upright pair reads the tangential one, and each pair is perfectly consistent with itself. The two images disagree with each other by hundreds of millimetres.

A third eye placed off both families — above and to the side of a level pair — sees the stick through rays that belong to neither image, and its ray to each point would miss the level pair’s reconstructed point by an amount the pair alone cannot see. A third ray is worth what its picture is worth measured what a third view buys in air, where every ray meets.

The measurement that follows adds a third camera at a stated offset above a level pair, reconstructs the stick from the pair as before, and asks how far the third camera’s rays miss the pair’s points: whether the disagreement grows with depth the way the pair’s own miss does between the exact rolls, how large an offset it takes before the third ray’s miss crosses the reading error at the tip, and whether the third camera’s miss is larger where the pair’s error in length is larger. If it is, a third eye turns the pair’s perfect agreement into a measurable disagreement, and a rig can detect that it is looking through water without being told.