Visual Plane
every picture states the point it is correct from
Through water and glass

The residual does not warn

Fit a straight line to the stick a stereo pair puts back and the fit looks best exactly where the reconstruction is least supported: the residual is 0.535 mm at a level baseline, where the two rays meet perfectly, and 0.038 mm at sixty degrees of roll, inside the band where they miss by more than a pixel covers. Over the same sweep the fitted line is 285 to 584 mm short of the stick's metre — fourteen thousand times its own residual at worst.
17 min read 5 figures Fitted, not assumedThe round tripWhat a projection destroys

Worth reading first: What a ray does at a surface · Two rays that do not meet.

The stick a stereo pair puts back found three sticks where there should have been one. Two eyes side by side reconstruct a submerged stick exactly as the sagittal image; two eyes one above the other reconstruct it exactly as the tangential one; and at every roll between, the two rays to a point miss each other by up to 3.13 millimetres — past the 2.85 a pixel covers at that range — so the reconstruction is a third stick that belongs to no real image. A level pair returns 715 millimetres of a one-metre stick, a rolled pair 551, an upright pair 416.

Nothing was fitted to any of those curves. The use a reconstruction is actually put to is fitting a shape — a straight line, a plane, a cylinder — and the question left open is what the compromise does to a fit rather than to a point. Specifically: does the residual warn? The rolled reconstruction is the one that belongs to no real image, so a reader with only the numbers in front of them would want its fit to look worst.

It looks best.

The fit, and the one number it reports

The measurement is plain. Reconstruct the stick at each roll of the baseline, fit a straight line to the reconstruction by least squares, and record three things: the residual the fit leaves, how far the fitted line’s direction is from the stick’s, and how far the reconstruction’s length is from the stick’s metre.

The fitted line is 12.7 degrees off the stick and 449 mm short, with a residual of 0.2 mmA one-metre stick leaning 30 degrees into water, the curve a stereo pair reconstructs with its baseline rolled 45 degrees, and the straight line least squares fits to that curve. The reconstruction is 551 mm long against a true 1000; the fitted line's direction is 12.70 degrees from the stick's; and the residual the fit reports is 0.21 mm. A reader deciding whether to trust the reconstruction is handed the last of those three numbers and none of the first two. The slider rolls the baseline.the stickreconstructedfitted straightbaseline rolled 45°residual 0.2 mm
Fig. 1 A one-metre stick leaning thirty degrees into water, the curve a stereo pair reconstructs with its baseline rolled forty-five degrees, and the straight line least squares fits to that curve. The slider rolls the baseline from level to upright.

At forty-five degrees of roll the fit’s residual is 0.214 mm. Its direction is 12.7 degrees from the stick’s, and the reconstruction it is fitted to is 449 mm short of the stick’s metre.

Only the first of those is available to a reader who does not already know where the stick is. Two tenths of a millimetre on a half-metre object reads as an excellent fit — a part in two and a half thousand — and it accompanies a shape that has lost nearly half its length and a good part of its lean.

The residual runs the wrong way

That would be forgivable if the residual at least moved with the trouble. The whole difficulty this field has been tracking is at the rolls between level and upright, where the rays miss and the reconstruction is a compromise no point in the water supports.

The rays miss by 3.13 mm at 45°, past what one pixel resolvesThe largest distance between the two rays to any point of the stick, as the pair's baseline rolls from level to upright. It is zero at both ends — a level pair reads the sagittal image and an upright pair the tangential one, both exactly — and rises to 3.13 millimetres in between. The level line is what one pixel of a nine-hundred-pixel camera covers at the stick's range, 2.85 millimetres: over 45° the miss exceeds it, so the reconstruction is a shape no point in the water supports rather than one a better matcher could sharpen. The kink of the reconstructed stick runs from 14.96 degrees to 9.58 over the same roll.0123020406080how far the pair's baseline is rolled from level, in degreeshow far the two rays miss each other, in millimetreswhat one pixel coverszero at level and at uprightworst 3.13 mm
Fig. 2 The rays’ own miss against the roll of the baseline, from the essay that found it: exactly zero at a level baseline and an upright one, and 3.13 mm at forty-five degrees, against the 2.85 mm one pixel covers at this range.

The miss is 0.000 mm at a level baseline, 3.132 mm at forty-five degrees, and 0.000 mm at an upright one. Those two zeros are not approximations: a level baseline puts both eyes in the family of planes the refraction is symmetric about, an upright one does the same by a different symmetry, and at both the two rays to a point genuinely meet. Between them there is a band in which the rays are further apart than the reading error can explain, and inside that band the reconstruction is not a point estimate with noise on it.

The fit looks best at 60 degrees of roll, 14 times better than where the rays meet exactlyTwo numbers a stereo reconstruction of a submerged stick could report, against how far the baseline is rolled from level. The rays' own miss is exactly zero at a level baseline and at an upright one — the two rolls whose reconstructions are real images of something, the sagittal stick and the tangential one — and rises to 3.132 mm at 45 degrees between them. The residual a fitted straight line leaves runs the other way: 0.535 mm at a level baseline, 0.214 where the rays miss most, and its MINIMUM of 0.038 mm at 60 degrees, inside the band where the rays are further apart than a pixel can explain. The fit looks best exactly where the reconstruction is least supported, because a residual reads the stick's own curvature and refraction bends the reconstruction least there.0123020406080roll of the baseline, degreesmillimetresthe residualthe rays' miss3.13 mm at 45°a one-metre stick, the baseline rolledthe two curves disagree about where the trouble is
Fig. 3 The residual a fitted straight line leaves, against the rays’ own miss, across the roll. The residual’s minimum sits at sixty degrees, in the middle of the band where the rays are furthest apart.

The residual does not merely fail to rise there. It falls. It is 0.535 mm at a level baseline, 0.401 at thirty degrees, 0.214 at forty-five, and reaches its minimum of 0.038 mm at sixty degrees — fourteen times smaller than at the level baseline, where the rays meet perfectly and the reconstruction is an exact image of something. Past sixty it climbs again to 0.371 at an upright baseline.

So the single number a pipeline reports is at its most reassuring inside the band where the reconstruction is least supported, and least reassuring at the two arrangements that are honest. A reader choosing a baseline by the residual would choose sixty degrees.

What the residual is actually reading

The behaviour is not perverse once the quantity is named. A residual measures how far a set of points departs from a straight line, and the reconstruction departs from a straight line because refraction bends it — a straight stick in water is a kink and a curve measured that bend directly, at 1.11° of further turning when the stick leans away from the eye and 7.62° when it leans toward.

Checked against the reconstruction’s own sag — how far its curve bows from the chord joining its ends — the residual is that sag divided by a constant. Across every roll from level to upright the ratio is 3.228, 3.228, 3.229, 3.232, 3.245, 3.195, 3.232, 3.236: the same number to one and a half per cent, over a sweep in which the residual itself changes by a factor of fourteen. The residual is a reading of the reconstruction’s curvature and nothing else.

And curvature is the wrong instrument here, because the compromise between two curves is also a curve. At sixty degrees of roll the two focal sticks’ contributions happen to cancel into something very nearly straight, so a straight line fits it beautifully. The straightness is a coincidence of the mixture, not evidence about the mixture.

The error is hundreds of times the residual

The residual’s other failing is one of scale, and it is the part a reader would act on.

A residual of 0.5 mm accompanies a length 285 mm short and a direction 14.3 degrees offWhat the fitted line actually gets wrong, beside what it reports. Across every roll the reconstruction is between 285 and 584 mm short of the stick's true metre and its direction is 10.4 to 14.3 degrees off, while the residual it leaves runs from 0.04 to 0.54 mm. The residual is between 532 and 15298 times smaller than the error in length alone, so a reader who accepts a fit on its residual is accepting a quarter of a metre of error on the evidence of a few millimetres.0.11101001000020406080roll of the baseline, degreesmillimetres, and ten times the degrees of tilt errorlength short bydirection ×10the residuala one-metre stick, thirty degrees into watershort by 285–584 mm throughout
Fig. 4 What the fitted line gets wrong, beside what it reports. Across every roll the reconstruction is between 285 and 584 mm short of the stick’s metre and its direction 10.4 to 14.3 degrees off, while the residual runs from 0.038 to 0.535 mm.

At every roll the reconstruction is at least 285 mm short of a one-metre stick and its fitted direction at least 10.4 degrees off. The residual over the same range is 0.038 to 0.535 mm. At a level baseline the error in length alone is 533 times the residual; at sixty degrees, where the fit looks best, it is 13,600 times it.

A reader deciding whether to believe the reconstruction is handed a number three or four orders of magnitude smaller than the smallest thing that is wrong. And the error is not scatter that more observations would average away — it is the same error every time, because it is what refraction does and not what the matcher does.

The control, which is the whole of the argument

Everything above could be an artefact of the triangulation, the fit, or the sampling. The way to know is to run the identical reading with the water taken away.

In air the same eyes return the stick to 1e-6 mm; in water it is 285 mm short at bestHow far the reconstructed stick falls short of its true metre, in water at three rolls of the baseline and in air. The air case is the control the whole reading depends on: the same eyes, the same triangulation, the same straight-line fit, and a reconstruction exact to 1.8e-13, 5.6e-14, 3.1e-12 m with a residual of 2.6e-13, 3.6e-13, 3.8e-13 m. Everything the water case gets wrong is the water's, and none of it is visible in the residual.in water, level baseline285 mmin water, rolled 45°449 mmin water, upright baseline584 mmin air, any baseline1e-6 mma one-metre stick, thirty degrees inthe control returns it exactly
Fig. 5 How far the reconstructed stick falls short of its true metre, in water at three rolls of the baseline and in air. The same eyes, the same triangulation, the same straight-line fit.

In air the stick comes back at its true metre to within 4×10−134\times10^{-13} m, with a residual of 3×10−133\times10^{-13} m and a fitted direction exact to a millionth of a degree, at every roll of the baseline. The rays meet, the reconstruction is straight, the fit is exact, and the residual is the arithmetic floor.

So every number above belongs to the water. The 285 millimetres is refraction’s, the 10.4 degrees are refraction’s, and the residual’s blindness to both is a property of what a residual is rather than of how this one was computed.

What a pipeline could report instead

The rays’ miss is the quantity that behaves. It is zero exactly where the reconstruction is an honest image of something and positive exactly where it is not, and it is available before the reconstruction is made — it is a property of the two rays, computed on the way to the midpoint.

The midpoint is a choice of ruler established that the midpoint of two skew rays is a convention rather than a measurement: it moves when the world is measured with a different ruler, by 0.203 mm under a threefold stretch and 1.503 mm under a projective frame, where the point of least reprojection error stays put to 10−1510^{-15} m. The finding here is the other half of the same complaint. Taking the midpoint does not only choose an answer arbitrarily; it discards the evidence that the question was ill-posed, and nothing downstream can recover it.

That gives a rule a pipeline can act on, and the rule is not “report the residual as well”. The miss is a statement about the rays and the residual is a statement about the shape, and this measurement shows they can point in opposite directions over the same sweep. A reconstruction reported with its residual alone has been laundered, and a reader who wants to know whether it is of anything at all needs a number the pipeline has already computed and thrown away.

Why the length is the thing that goes wrong

The residual’s blindness would matter less if the error it is blind to were small. It is not, and it is worth saying which error it is, because the reconstruction does not fail in the way a noisy one would.

A noisy reconstruction is scattered about the truth. This one is not scattered at all: it is a smooth curve lying entirely inside the true stick, missing it by more the deeper the point. Every reconstructed point is nearer the surface than the point it stands for, because refraction bends a ray away from the normal on the way out and a straight-line back-projection therefore stops short. A point under water has two depths is the reading of that for a single point; along a whole stick it accumulates into a shortening.

So the failure is a systematic contraction along the line of sight, and the numbers say how severe: 715 mm returned at a level baseline, 551 at forty-five degrees of roll, 416 at an upright one, against a true 1,000. The fitted line inherits it exactly, because a least-squares line through a contracted curve is a contracted line.

That is why a residual cannot see it. A residual is a measure of how far the points are from some line; a contraction moves the points along the line they are already near, which costs a residual nothing. The one error a straight-line fit is structurally incapable of noticing is the one this geometry produces.

What a reader should take from it

Three readings, in the order they would be used on a real reconstruction.

A small residual is not evidence of a good reconstruction. It is evidence that the points lie near a line, which they can do while being in the wrong place, the wrong size and the wrong direction. Here it is three to four orders of magnitude smaller than the error it accompanies.

A residual is not even monotone in the trouble. It is 0.038 mm at the roll where the rays are furthest from meeting and 0.535 where they meet exactly. A reader comparing two reconstructions by their residuals would prefer the worse one, which is a stronger failure than mere insensitivity.

The same caution applies to every shape a pipeline fits. A plane fitted to a submerged floor, a cylinder to a pipe, a sphere to a float — each reports a residual, and each residual is a reading of how far the reconstruction departs from that shape rather than of whether the reconstruction is of anything. A point under water has two depths is the reason to expect the same failure in each: the two focal surfaces are properties of the geometry and not of the object, so whatever shape is fitted inherits the contraction and reports only its own curvature.

And the quantity that behaves was already computed. Every triangulation that takes a midpoint has the two rays’ closest approach in hand at the moment it takes it. Reporting it costs one number per point, and the stick a stereo pair puts back found it 3.13 mm where a pixel covers 2.85 — measurable, and not in any report.

What this does not settle

A straight line is the friendliest shape there is. Fitting a plane or a cylinder gives a compromise more ways to hide and a residual more ways to be small, and nothing here says the shape matters. It very likely does, and in the direction that makes this worse.

The miss is a small number too. It is 3.132 mm at its worst against the 2.85 mm one pixel covers at this range, so the band in which it is unambiguous is narrow — roughly thirty to sixty degrees of roll. Outside that band the miss and the reading error are the same size, and the warning this essay recommends is available only where the trouble is largest.

Water is the only medium here. Everything is computed at water’s index against air, and the whole effect scales with how far that index is from one — a point under water has two depths is the essay about what the two focal surfaces are, and a weaker medium moves them together. Whether the residual’s minimum survives a weaker refraction is not measured.

The stick is in the vertical plane through the eye, leaning away. That arrangement is what gives the two exact rolls their symmetry and what makes the control a control. A stick turned out of that plane has no exact roll at all, so there is nothing for the residual to fail to find, and the comparison this essay rests on does not exist there.

And the fit is unweighted. Every point of the reconstruction counts the same, which is not what a careful fit does — a third ray is worth what its picture is worth is about weighting each observation by what it is worth, and a fit that weighted points by their rays’ miss would be using exactly the evidence this essay says is discarded.

Three numbers, one of them available

A straight line fitted to the stick a stereo pair puts back reports a residual of 0.535 mm at a level baseline, 0.214 at forty-five degrees of roll and a minimum of 0.038 mm at sixty — which is inside the band where the two rays are further apart than a pixel can explain. The fit looks fourteen times better where the reconstruction belongs to no real image than where it is exact. What the residual measures is the reconstruction’s own curvature, at a constant 3.23 times its sag across the whole sweep, and a compromise between two curves is also a curve.

Meanwhile the fit is 285 to 584 mm short of a one-metre stick and 10.4 to 14.3 degrees off its direction: between 533 and 13,600 times the residual it reports. In air the same eyes, the same triangulation and the same fit return the stick exactly, so all of it is the water’s.

The one quantity that is zero at the two arrangements whose reconstructions are real images and positive at every arrangement between them is the rays’ own miss — and a reconstruction that takes midpoints computes it and throws it away.

Still open: whether a fit can be weighted by the miss

The last two sections point at the same repair from opposite ends. The miss knows which points of a reconstruction are supported by rays that meet and which are not, and the fit treats every point alike.

A fit weighted by the miss would put its trust where the rays agree. On a rolled baseline that is not a uniform discount: the miss varies along the stick, largest where the ray geometry is most skew and smallest near the surface where the stick enters, so a weighted fit would be pulled toward one end of the reconstruction rather than shrunk overall.

The measurement that follows fits the same line with each point weighted by the inverse of its own rays’ miss, sweeps the roll from level to upright, and asks three things: whether the weighted fit’s direction is closer to the stick’s than the unweighted one’s, whether its residual now falls where the miss falls — which would make the residual a warning at last — and what happens at the two exact rolls, where every weight is infinite and the weighting has to be regularised before it means anything. If the weighted residual tracks the miss, a pipeline can be given back the warning it discards for the price of carrying one extra number per point; if it does not, then the miss has to be reported on its own and the residual has to be labelled as what it is, which is a reading of curvature.

Where this goes next

eyethe stickcorrect from 18 cm, at 160 mm widerays miss by 3.13 mm The stick a stereo pair puts back Two eyes side by side reconstruct a submerged stick exactly as the sagittal image — kinked 14.96° — and two eyes one above the other exactly as the tangential one, kinked 9.59°. Roll the baseline between them and the two rays to a point miss each other by up to 3.13 millimetres, past the 2.85 a pixel covers at that range, and the reconstruction is a third stick that is neither — 551 millimetres of a one-metre stick, with its tip at 0.405 metres against a true 0.866. eyethe sticktwo eyes side by sidea nodding headcorrect from 18 cm, at 160 mm widekink 14.96° · turn 4.29° A straight stick in water is a kink and a curve The bent stick is described as one kink at the surface. Traced point by point, the image two level eyes see leaves the surface 14.96° off a stick leaning 30° — the same whichever way it leans — and then keeps turning, by 1.11° when it leans away from the eye and 7.62° when it leans toward it. A photograph from the eye shows the kink and almost none of the curve, 1.63 px over 166 px, and when the stick leans straight toward or away from the eye it shows neither: the picture is one straight line. the needle: along the rays' bisectorreprojection minimum, both framesmidpoint, world framemidpoint, changed framemidpoint moved 0.203 mm · minimum moved 1.0e-15 mtruth 15.5 mm along the needle The midpoint is a choice of ruler Two photographs do not change when the world is measured with a different ruler, so an answer that belongs to the photographs cannot change either. The midpoint of two skew rays does: a threefold stretch moves it 0.203 mm and a projective frame 1.503 mm, while the point that minimises reprojection error stays put to 10⁻¹⁵ m. Both are 15.5 mm from the truth, which is the part a choice of route does not touch.

Shares its objects with

Essays that name at least two of the same things, and that neither author linked.

Named objects

A flat tag is an object no other essay names yet.

Apparent-depthAstigmatismBaselineleast squaresMidpointRefractionResidualskew raysStereo pairTriangulation