The midpoint is a choice of ruler

Two rays that do not meet ended with three answers to one question. Handed two marks read to a pixel, whose rays pass each other 2.77 mm apart, a reconstruction can return the midpoint of the shortest segment between the rays, the null vector of a stacked linear system, or the point whose projections land nearest the marks. The essay called all three defensible, preferred the third because it is measured in the units the error occurred in, and moved on.

That preference was argued from units. There is a stronger test available, and it does not involve the truth at all. It asks which of the three answers is a property of the two photographs, and it has a definite result: one of them is, and the midpoint is not.

A test that needs no ground truth #

Take the two cameras and the two marks, and describe the world in different coordinates. Rotate the axes, rescale them, stretch them along some direction, or do something more drastic that sends a distant plane to infinity. Rewrite each camera’s matrix to match — the new matrix is the old one times the inverse of the change of frame — and project the world through it.

The pictures do not change. Not approximately: every point lands on exactly the pixel it landed on before, because the change of frame and its inverse cancel inside the product. Two photographers standing in one courtyard, one of whom measures in metres along the walls and the other in some sheared and stretched system of their own, take identical photographs, and nothing in the photographs says which ruler was in use.

So an answer that is computed from the photographs alone, and that is genuinely about them, must survive the change. Triangulate in the new frame, map the result back into the old one, and it must land where the triangulation in the old frame did. An answer that moves has been computed from the photographs plus the ruler, and the ruler is not in the photographs.

That test is cheap to run and it does not care which answer is closer to the truth. It separates quantities that belong to the data from quantities that belong to a choice made about the data, a distinction that recurs wherever a scene is recovered, and which seven numbers no picture can name drew for a whole reconstruction.

The control: a ruler that only rotates and rescales #

The first frame is a similarity: a turn of forty degrees about a skew axis, a scale of two and a half, and a shift. It is the frame change that a careless reader of “coordinates are arbitrary” has in mind, and both answers survive it.

The reprojection minimum moves by 1.3 × 10⁻¹⁵ m, which is arithmetic. The midpoint moves by less than a hundredth of a picometre, which is also arithmetic. Neither has noticed anything.

That is worth having before the interesting cases, because it says what the midpoint is made of. It is found by minimising a distance: the point closest to both rays, in metres. A distance changes under a similarity only by the uniform scale, and minimising a quantity that has been multiplied by two and a half everywhere finds the same point. Angles do not change at all. So a construction built from lengths and right angles — the common perpendicular is defined by two right angles and its midpoint by a ratio of two lengths — is untouched by any frame that preserves both.

The control therefore does its job twice. It checks the computation, since an error in the frame algebra would move both points here, and it establishes the boundary of the claim: the midpoint is a Euclidean construction, and it is safe in every frame that is Euclidean up to scale.

A ruler stretched along one direction #

The second frame stretches lengths by three along one direction — nearly horizontal, turned seventy degrees from the axis across the courtyard — and leaves every perpendicular direction alone. It is the simplest frame that is affine but not a similarity. Parallel lines stay parallel and midpoints of segments stay midpoints, but a right angle between the stretched direction and any other stops being one.

Now the two answers part. The midpoint computed with the stretched ruler, mapped back into ordinary metres, lands 0.203 mm from the midpoint computed with the ordinary one. The reprojection minimum, computed in both, moves 1.0 × 10⁻¹⁵ m.

The reason is the common perpendicular’s own definition. “The shortest segment joining the two rays” is shortest in some measure of length, and the stretched frame measures length differently: a segment leaning toward the stretched direction counts as three times as long as it did. The segment that minimises the stretched length is a different segment, with a different midpoint, and mapping it back does not restore the old one, because what was minimised was different.

The reprojection minimum has no such dependence. It is the point whose projections sit nearest the two marks, measured in pixels, in the pictures. The frame change rewrites the cameras so that every point projects to the same pixel as before; the objective is therefore the same function of the world point, merely written in other coordinates, and its minimum is the same world point. Nothing about the answer consulted a ruler.

How far the stretch can push it #

A threefold stretch is one setting. The natural next question is whether the midpoint keeps moving as the ruler is distorted further, and the sweep says it does not — at least not for a stretch.

At ×1 the frame is the identity and nothing moves. By ×2 the midpoint has moved 0.171 mm; by ×5, 0.220 mm; by ×20, 0.228 mm. The curve flattens because a very strong stretch makes the stretched direction’s component dominate every length, so the shortest segment in that measure converges to the one that is shortest along that direction alone. The ruler’s distortion saturates, and so does its effect.

That ceiling is a property of affine frames, and it is worth being careful not to promote it. A stretch leaves parallels parallel and keeps the plane at infinity where it was. The frame in which a reconstruction from uncalibrated photographs actually comes out does neither.

A frame that sends a plane to infinity #

The third kind of frame is projective: it takes a plane in the world — here a vertical plane three metres to one side — and sends it to infinity, bending every straight line’s spacing as it goes while keeping every line straight. Nothing about the two photographs changes under this frame either. The cameras are rewritten, every point lands on its old pixel, and a reconstruction handed only the pictures has no way to prefer this frame to the ordinary one.

Here the midpoint moves 1.503 mm, more than half the width of the gap it is the middle of. The reprojection minimum moves 1.1 × 10⁻¹⁵ m.

And there is no ceiling. Bring the vanishing plane in from eighty metres to three and the midpoint’s displacement grows the whole way — 0.022 mm, 0.108 mm at twenty, 0.269 mm at ten, 1.003 mm at four, 1.503 mm at three. The sweep stops at three metres only because a nearer plane would pass between the courtyard and one of the cameras, and a frame that splits the scene from its own eyes is answering a different question; those settings are refused rather than drawn.

A projective change of frame does not preserve the midpoint of a segment, let alone a shortest distance, so there is nothing in the construction for it to leave alone. The midpoint of the common perpendicular in a projective frame is a point defined by metric notions that the frame has scrambled, and the scrambling grows without limit as the plane sent to infinity approaches the scene.

Why the projective case is the one that matters #

It would be easy to file all of this as a curiosity: nobody measures a courtyard with a ruler whose vanishing plane is three metres away.

But a reconstruction does, routinely, before it knows better. Two views give shape and no size set out the sequence: with nothing but correspondences, two photographs determine the scene up to an arbitrary projective transformation of space, fifteen free numbers, and only knowledge from outside the pictures upgrades that to affine, then metric, then Euclidean. A reconstruction from uncalibrated photographs is, by construction, expressed in a frame of exactly the kind the third figure uses — and nobody chose which one.

In that frame the midpoint is not a slightly worse answer. It is an answer to a question the reconstruction cannot yet ask, because “the point closest to both rays” presupposes a notion of distance, and the projective stage has none. Computing it anyway returns a number that depends on which of infinitely many equally valid frames the algebra happened to land in, and a second run of the same software from a different starting frame would put it somewhere else.

The reprojection minimum can be computed at every stage of that sequence, including the lowest, because its objective lives in the pictures. That is a stronger reason to prefer it than the one the earlier essay gave. Measuring the error in pixels is appropriate because pixels are where the error happened; minimising it in pixels is necessary because pixels are the only units the reconstruction possesses until something from outside supplies more.

The same distinction reappears at every stage of the sequence of assumptions and what each costs, which measured how the recovered quantities degrade as they climb. There, a cross-ratio read in the picture was immune to errors in the recovered map because it never went through the map. Here, the reprojection minimum is immune to the choice of frame because it never goes through the frame. Both are the same fact: a quantity computed where the data lives is safe from everything done to the data afterwards.

Which way the midpoint moves #

The displacement has a direction as well as a size, and the direction is the one two rays that do not meet described without measuring.

That essay argued that a triangulated point lives in a long thin sliver shaped like a needle, pointing back at the cameras, because the two cones of possible rays cross at a shallow angle. The pictures are least sensitive along the needle’s length: sliding a point along it barely changes where it projects.

The frame figures are drawn in that sliver’s own coordinates, and the midpoint moves mostly along the needle. Under the threefold stretch, of the 0.3 mm between the moved midpoint and the reprojection minimum, 0.29 mm lies along the needle and under a tenth of a millimetre across it. The distortion pushes the answer in the direction the data already failed to pin down, which is the only direction it could push it far without the pictures objecting.

That also explains why a pixel-based diagnostic sees almost nothing. The reprojection error of the midpoint computed with the stretched ruler is 0.1405 px, against 0.1401 px for the minimum itself: a disagreement in the fourth decimal place of a pixel. Under the projective frame at three metres it rises to 0.1433 px. A reconstruction reporting its reprojection error would describe both midpoints as equally good fits, while one of them sits a millimetre and a half away from the other, along the axis where a millimetre is cheap in pixels and expensive in the world.

The same stretch on a different correspondence #

The size of the effect is not a constant of the rig. It depends on the correspondence, because the needle’s direction and the rays’ crossing angle both do.

Correspondence 11’s rays cross at 25.5°, against 19.9° for correspondence 7, and they pass 2.47 mm apart rather than 2.77. By the reasoning about needles it should be the better-placed point, and it is. Yet the same stretch moves its midpoint 0.675 mm, more than three times as far. What decides the displacement is not how well the point is triangulated but how the stretched direction sits against the needle: a stretch aligned with the direction the data leaves loose has most to work with.

A corner at the far end of the courtyard, whose rays pass only 0.67 mm apart because it was read more luckily, moves 0.017 mm under the same frame. Every one of the three is still a millimetre-scale answer being moved by a sub-millimetre amount, and every one of their reprojection minima stays where it was to the fifteenth decimal place.

What invariance does not buy #

It would be a misreading to conclude that the reprojection minimum is therefore the more accurate answer, and the numbers say plainly that it need not be.

For correspondence 7 the true world point lies 15.5 mm along the needle from all three answers. The midpoint computed in metres is 15.57 mm from it; the reprojection minimum is 15.65 mm from it. The frame-dependent answer happens to be the closer of the two, by eight hundredths of a millimetre, and for correspondence 11 the ordering reverses. Neither route is systematically nearer the truth on a single correspondence, because both are choosing a point inside a region whose size is set by the reading, and the reading put the truth fifteen millimetres from the middle of that region.

That is the proportion worth keeping. The frame moved the midpoint by 0.2 mm under an affine ruler and 1.5 mm under a severe projective one. The misreading of a single pixel put every candidate 15 mm from the truth. Choosing a better route is a correction at the level of a tenth of the error; the reading is the error.

So the argument for the reprojection minimum is not accuracy. It is reproducibility. Two reconstructions of the same photographs, computed in different frames by different software, will agree on the reprojection minimum to the last bit and will disagree on the midpoint by an amount that depends on nothing physical. An answer that two honest computations disagree about is not yet an answer, whatever its distance from the truth happens to be.

Two routes to the minimum, and what their agreement proves #

The reprojection minimum in these figures is found by a Gauss–Newton descent in space, started from the midpoint and refused if it has not converged. A second route finds the same point without leaving the pictures: every matching pair of epipolar lines is one member of a pencil, so the corrected marks nearest the measured ones are found by searching a single angle, and their rays then meet exactly.

The two routes agree to 8.5 × 10⁻⁹ m on correspondence 7 and report the same reprojection error, 0.1401 px. That agreement checks the implementation of the minimum. It says nothing about frames, and it should not be quoted as though it did: both routes minimise the same objective, and two computations of one objective agreeing is a statement about arithmetic. The frame test is the separate measurement, and it is the one that could have failed.

The distinction is the one a wrong match is not a small error made about residuals, turned toward a different object. A residual measures agreement between a model and its own fit. Two routes to one minimum measure agreement between two implementations. Neither measures whether the quantity computed is the quantity wanted, and only a perturbation the answer ought to survive — here, a change of frame that leaves the data untouched — does that.

The rule this fits #

The midpoint is not a bad construction. It is a construction that presupposes a metric, and it gives exactly the right answer to a question that includes the metric: the point nearest both rays, as measured by this ruler. The failure is only in forgetting that the ruler is part of the question.

That is the same shape as depth is a reciprocal, where a symmetric error bar is the right answer to a question about a linearised model and the wrong one about the rig, and as an angle is a cross-ratio, where an angle read off a photograph is available only once something has supplied what a right angle is. In each case a quantity looked like a property of the picture and was a property of the picture together with an assumption, and the way to tell was to change the assumption and watch what moved.

The practical form is short. Triangulate in the pictures’ own units, so that the answer survives whatever frame the reconstruction is expressed in. Compute the gap between the rays as well, because that essay showed it reports what the reprojection error cannot. And when the metric is known — when the reconstruction has reached the Euclidean stage — the midpoint is again a legitimate answer, provided the frame it is computed in is the one the metric belongs to.

Still open: how more rays than two should be weighed #

Everything above concerns two rays. The case that matters in practice has more, and it changes the midpoint’s failure from a dependence on the frame into a dependence on which cameras are far away.

With three or more rays, the point nearest all of them in metres is a least-squares point in which every ray counts equally per metre of miss. A ray from a distant camera, whose angular reading error spans more metres at the point, therefore pulls as hard as a ray from a near one whose reading was worth far more. The reprojection minimum weights each ray by what its own picture says about it, which is roughly one over the distance. What remains is to measure that difference on a rig of three eyes in which one is moved progressively further back: how far the metric point is dragged toward the worst-placed camera, against the reprojection minimum, as the distant eye’s share of the error grows — and whether, as the needle picture predicts, a third ray from a different direction shortens the sliver more than any number of rays from nearly the same one.