Two rays that do not meet

Every account of stereo reconstruction contains a sentence like this one: the point is found by intersecting the two rays.

It is a good sentence for conveying the idea and it describes something that does not happen. Two lines in three-dimensional space, chosen at random, do not intersect. They are skew — they pass one another at some distance, and that distance is generically not zero. Two rays that come from a real pair of measurements are exactly such a pair of lines, and the reconstruction has to do something other than intersect them.

What it does instead is a choice, and different choices give different answers.

The exact case, and why it is the unusual one #

With correspondences placed exactly — the same world point projected into two cameras, with no rounding anywhere — the two rays do meet. The figure measures the gap at 2.2 × 10⁻¹⁶ m, which is the arithmetic’s floor and not a length.

That has to be true, and it is worth saying why rather than treating it as reassurance. Both rays pass through the world point that produced both marks. Two lines through a common point meet at that point. The exactness is not a property of the algorithm; it is the definition of the input.

So the exact case is a check on the machinery rather than a result. What it verifies is that the ray construction, the camera basis, the principal point and the focal length are all consistent — if any of them were wrong, the rays would miss even on exact input, and the miss would be a bug rather than a measurement.

The interesting case is the one where the input is what a real measurement produces. Round the same two marks to a whole pixel, which is what reading a point off a picture does, and the rays miss by 2.77 mm at a range of 7.22 m.

Where a millimetre of gap comes from #

A pixel is not a length in the world; it becomes one at a distance. At 7.22 m with the field’s cameras, one pixel subtends roughly half a centimetre, so displacing a mark by half a pixel moves its ray by a couple of millimetres at that range. Two marks each displaced independently give rays that miss by a comparable amount, and 2.77 mm is what falls out.

The number is therefore unsurprising in magnitude. What is worth noticing is that it exists at all, and that nothing downstream is obliged to mention it.

A reconstruction routine handed those two marks returns a point. The point is somewhere near the middle of the gap, it reprojects to within a fraction of a pixel of both marks, and every summary statistic looks healthy. The 2.77 mm has been absorbed silently, and the only trace of it is in a quantity that most implementations never compute.

Two routes, two answers #

There are two natural things to do with a pair of skew rays, and this site implements both because two independent routes to one number is its standing habit.

The midpoint of the common perpendicular. There is a unique shortest segment joining two skew lines. Take its midpoint. This is geometric, it is easy to picture, and it returns the gap as a by-product — which is why the figure uses it.

The direct linear transform. Write each camera’s projection as two linear equations in the unknown point’s homogeneous coordinates, stack the four, and take the null vector. This is what triangulate does, it generalises immediately to any number of views, and it is the standard method.

On exact input the two agree, because both return the intersection when one exists. On real input they do not, and the disagreement is not a bug in either.

The reason is that they minimise different things. The midpoint minimises a distance in space — it is the point that is closest, in metres, to both rays. The direct linear transform minimises an algebraic quantity built out of the homogeneous coordinates, which weights points differently according to their depth and their position in the frame. Neither of them minimises the quantity that a photogrammetrist would actually want minimised, which is the reprojection error in pixels — the distance in each picture between the mark and where the reconstructed point lands.

Three quantities, three answers, all defensible. The one that is usually wanted is the third, because it is the only one measured in the units the errors actually occurred in: the marks were misread by pixels, so a point that explains the marks well is one whose reprojection is off by few pixels, not one that is near two lines in metres.

That is exactly what a bundle adjustment minimises, over every point and every camera at once, and it is the reason the many-view field’s numbers are quoted in pixels throughout.

Why the gap is worth reporting #

A practitioner looking at a reconstruction has one number readily available, which is the reprojection error. It is easy to compute, it is in sensible units, and it is what most software reports.

The gap is a different quantity and it says something the reprojection error does not.

Consider a point very far away, seen by two cameras a short distance apart. The two rays are nearly parallel. A small angular error in either of them moves the crossing region enormously along the line of sight, and the resulting gap can be large — while the reprojection error stays small, because moving the reconstructed point along the ray barely changes where it lands in either picture.

That is the near-degenerate case the depth field is about, seen from the triangulation side. A reconstruction with excellent reprojection error and a large gap is a reconstruction whose points are well determined across the line of sight and poorly determined along it. The reprojection error cannot say that, because it is measured in the pictures and the uncertainty is in the direction the pictures are least sensitive to.

The gap can. It is measured in the world, along the direction the pictures are worst at, and it is large exactly when the configuration is bad. That makes it a diagnostic for the failure mode that a residual is structurally unable to see.

This is the third instance on this site of a residual that could not report the thing it was being asked about. The conformality measurement gave a perfect score to a surface that fails, because it was evaluated along the one tangent basis where it cannot fail. The plumb-line lens fit returned a residual of 0.68 px with a coefficient 22% wrong. And here a reprojection error of a fraction of a pixel accompanies a reconstruction that is uncertain by metres along the view direction. In each case the residual is correct and is answering a different question.

The shape of the uncertain region #

It is worth being concrete about the region a triangulated point actually lives in, because “uncertain by a few millimetres” suggests a ball and the region is nothing like one.

Each mark, read to a pixel, does not determine a ray — it determines a thin cone. The reconstructed point is somewhere in the intersection of two such cones, and the intersection of two thin cones crossing at a shallow angle is a long thin sliver, elongated along the bisector of the two view directions.

Its width across the line of sight is set by the mark precision times the range, and at 7.22 m with these cameras that is a few millimetres. Its length along the line of sight is set by the same precision divided by the crossing angle, and the crossing angle is small — which is why the depth interval is far larger than the lateral one and why it grows quadratically while the lateral one grows linearly.

So a triangulated point should be pictured as a needle pointing back at the cameras, not as a dot with a tolerance. The needle’s aspect ratio is roughly the range divided by the baseline, which for a 2.6 m baseline at 7.22 m is about three, and for a 65 mm baseline at 10 m is about a hundred and fifty.

That picture explains several things at once. It explains why the two triangulation routes disagree — they pick different points along a long thin region, and the region’s length is the disagreement’s scale. It explains why the reprojection error stays small while the depth is poorly known: sliding along the needle is exactly the motion the pictures are least sensitive to. And it explains why a third view, taken from a different direction, helps far more than a more careful reading of the first two: a third cone crossing the needle sideways cuts its length, and no improvement in the first two marks does that.

The plane the figure is drawn in #

A note on the drawing, because the choice carries an argument.

Two skew rays and the segment joining them are a three-dimensional configuration, and drawing it in perspective would produce a picture in which the gap is foreshortened by an unknown amount — which on a site about perspective would be a poor joke.

So the figure works in the plane containing both rays’ closest approach: the two rays are projected into it, and the common perpendicular appears at true length. What the reader sees is the actual gap, at a stated scale, rather than a view of it.

The near-parallelism is also visible, and it is the point. The two rays in the figure cross at a shallow angle, which is what a 2.6 m baseline at 7 m range gives, and the shallowness is why a small displacement of either mark opens a millimetre-scale gap. A figure drawn at a wide crossing angle would show two lines meeting decisively and would make the gap look like a curiosity.

The correction that is not available #

There is a repair that looks obviously right and is worth explaining, because it is available in principle, is used in practice, and is not what the figure does.

The two marks, as read, are inconsistent — no world point projects to both. But there is a nearby pair that is consistent: move each mark by the smallest amount that puts them onto a matching pair of epipolar lines, and the corrected pair does have an exact intersection. Triangulating the corrected marks gives a point with zero gap, and the total correction applied is the reprojection error of that point, which is the quantity worth minimising.

That procedure is optimal in a precise sense, given a sensible noise model, and it has a closed form. It is the right thing to do when a reconstruction is what is wanted.

It is the wrong thing to do here for one reason: it makes the gap disappear by construction. A figure that corrected the marks first and then reported that the rays met would be reporting the correction rather than the geometry, and would look exactly like a figure demonstrating that triangulation is an intersection. The measurement this essay is about is how far the uncorrected marks were from consistency, and correcting them is precisely the operation that destroys it.

This is a case of a rule this site applies throughout: do not evaluate a claim at the one input where it cannot fail. The conformality test fell into that trap by differencing along the one tangent basis that stays perpendicular, and the cross-ratio check fell into it by using four consecutive divisions. Triangulating epipolar-corrected marks and reporting a zero gap would be the same mistake with a different subject.

Both versions are worth having, and the ordering matters. The gap says what the measurement contained; the correction says what to do about it.

What triangulation cannot do #

Three limits, stated because they are all routinely elided by the intersection metaphor.

It cannot tell a bad match from a distant point. Both produce a large gap. A correspondence that is simply wrong and a correspondence that is right but at the edge of the rig’s range look similar from inside the triangulation, and separating them needs information from outside — a third view, a prior on where things are, or a consistency check across many points.

It cannot recover scale. The gap is in the same arbitrary units as everything else in the reconstruction, so “2.77 mm” is only a length because this figure’s cameras were built with a stated baseline in metres. In a reconstruction from two arbitrary photographs it would be a number times an unknown factor.

And it cannot be improved by choosing a better point between the rays. Once the two rays are fixed, every method is choosing a point in a region whose size is set by the gap. Being cleverer about which point moves the answer within that region and does not shrink it. Shrinking it means better marks, a longer baseline, or more views — that is, changing the input rather than the estimator.

The habit this belongs to #

The metaphor that fails here fails in a specific way, and it is a way this site keeps meeting.

“Intersect the two rays” describes an operation on idealised objects and is used to describe an operation on measured ones. That is the same move as “run a line to the vanishing point” — correct for the ideal, and silent about what a real construction does when the lines do not concur. It is the same move as “the shadow is the projection of the solid”, which is true of the outline and produced a bow-tie when it was implemented as a list of vertices.

In each case the repair is the same: implement the statement that is actually true of the measured objects, and report the quantity that says how far from the ideal they are. Here that means computing the common perpendicular rather than an intersection, and printing its length.

The gap is 2.77 mm. That is not a failure of the reconstruction and it is not noise to be hidden. It is the measurement, and a routine that does not compute it is discarding the only quantity that reports what the pictures could not agree on.

There is one further reason to keep it, which has nothing to do with diagnosing configurations and everything to do with knowing when the model has stopped applying. Everything above assumes both rays are straight lines through their cameras’ centres — that each picture is a projection through a point. Where that assumption fails, the rays miss by amounts that have nothing to do with reading precision, and the gap is the quantity that notices.

The refraction field measured exactly this for a single view: rays continued into water, fitted to a common point, miss by 8.3 mm where the same fit with the water removed lands on the eye exactly. That is the same measurement as this essay’s, made on one camera’s own bundle rather than across two, and it is three times larger than the gap a whole pixel of misreading produces here. A pair of photographs taken through an aquarium would therefore show a triangulation gap dominated by the water rather than by the marks — and the reprojection error, once again, would report nothing at all.