A rig is right on one surface

The two rungs before this one are about a mistake: a camera bolted to its tripod by the wrong point, when the right point was available and a rail would have found it. The rig is not a mistake. Put several cameras on a ball and point them outward and they have several entrance pupils, several centimetres apart, and nothing can be done about it — two lenses cannot occupy the same place.

So the question changes. It is no longer how badly is this mounted but which depth should the stitch be right at, since it cannot be right at all of them.

The residual is a disparity, and disparities are reciprocal #

Two cameras a baseline $b$ apart, both looking the same way, image a point at depth $z$ at positions differing by $f b / z$ pixels. That is the stereo disparity the depth field is built on, and it is what a rig’s stitch is fighting.

A warp computed to register depth $z_0$ is a shift of $f b / z_0$. Apply it and what is left over is

$$\Delta(z) \;=\; f\,b\left(\frac1z - \frac1{z_0}\right)$$

which is zero at $z_0$ and nowhere else. The picture is correct on one surface — the sphere or cylinder of radius $z_0$ around the rig — and every feature off it is drawn twice, or cut in half where the seam runs through it.

The reciprocal is doing the work in everything that follows. A depth range from a metre and a half to thirty metres is a factor of twenty in distance and a factor of twenty in disparity, and the near end dominates so heavily that any choice made by looking at the metres will be wrong.

The best depth is the harmonic midpoint #

Ask for the choice of $z_0$ that makes the worst residual over a range $[z_n, z_f]$ as small as possible. The residual is monotone in $1/z$, so the worst is always at one end or the other, and the minimum of the worse of the two is where they are equal:

$$\frac1{z_n} - \frac1{z_0} \;=\; \frac1{z_0} - \frac1{z_f} \qquad\Longrightarrow\qquad z_0 \;=\; \frac{2 z_n z_f}{z_n + z_f}$$

which is the harmonic mean of the two ends. For a scene from one and a half metres to thirty, that is under three metres — a fifth of the way along the range in metres, and exactly halfway along it in disparity.

The search in that figure is not decoration. The closed form above is three lines of algebra and three lines of algebra can be wrong; a sweep over four thousand candidate depths that lands on the same number is what makes it a measurement. assertTheStitchingDepthIsTheHarmonicMidpoint requires the search’s minimum to agree with the formula to two parts in a thousand, which is the resolution of the sweep rather than of the claim.

The penalty for the obvious choice, in closed form #

The arithmetic midpoint is what a person picks. It is wrong, and by how much has an exact answer:

$$\frac{\text{worst at the arithmetic midpoint}}{\text{worst at the harmonic one}} \;=\; \frac{2 z_f}{z_n + z_f}$$

For a range of one and a half to thirty metres that is a shade over one point nine — nearly twice as bad. And as the far end runs away it tends to exactly two and never exceeds it, so the penalty for the natural choice is bounded, and the bound is a factor of two.

That bound is the useful half. A rig whose stitching depth was set by somebody eyeballing the middle of the room is not catastrophically wrong; it is within a factor of two of the best it could have been, forever, whatever the scene. Which is worth knowing because it says where not to spend effort: the depth is worth choosing well and it is not worth choosing perfectly, and the thing that actually matters is the next paragraph.

And the floor, which is why the choice is small #

Put the harmonic midpoint back into the residual and the worst case comes out as

$$\tfrac12\, f\, b \left(\frac1{z_n} - \frac1{z_f}\right)$$

— half the disparity spread of the whole depth range. That is the best any choice of $z_0$ can achieve, and it is not zero. On an eighty-millimetre baseline over a range from one and a half to thirty metres it is tens of pixels, which is a visible doubling.

So the optimisation is real and it is not a repair. assertNoStitchingDepthIsRight exists to stop the previous section being read as one: it requires the best achievable worst case to equal the closed form for the floor, and requires that floor to be pixels rather than nothing, and then requires it to vanish exactly when the baseline does. Without the third the floor could be an artefact of the fit; without the second the whole essay could be read as a tuning guide.

The floor is the near end, and it has a hyperfocal distance #

Two substitutions into the floor turn it into rules a rig operator can use, and the second of them makes the whole problem an old one in new clothes.

Send the far end to infinity. Almost every real scene has a distant background, and then $1/z_f \to 0$, so the harmonic midpoint becomes $z_0 = 2z_n$ and the floor becomes

$$\frac{f\,b}{2\,z_n}.$$

Register at twice the distance of the nearest object, and the residual is set by that object alone. The far end has dropped out of both expressions — a scene running to thirty metres and one running to the horizon get the same answer, which is why the essay’s own 1.5-to-30 case gives 2.86 m against the 2.00 m rule and a floor within five per cent of it.

Put numbers in. An 80 mm rig at 950 pixels of focal length with the nearest thing at a metre and a half has a floor of $950 \times 0.08 / 3 = 25$ pixels, which is the tens the section above reports. Inverted, holding the floor to one pixel needs

$$z_n \;>\; \frac{f\,b}{2\tau} \;=\; 38 \ \text{metres}.$$

A rig with an 80 mm baseline cannot stitch anything nearer than about forty metres to within a pixel, whatever is done to the warp, the software or the depth chosen. That is the honest scope of the instrument, and it is why a rig’s output looks flawless outdoors and doubles everything in a room.

One thing does not help, and it is the thing that helped on the previous rungs. As an angle the floor is $b/2z_n$ radians, with the focal length cancelled — so a longer lens magnifies the picture and the error equally and buys nothing at all. The pivot error of a hand-turned panorama falls with a longer lens because a longer lens makes a shorter frame; a rig’s baseline is not an angle and has no such relief.

And the acceptance zone is a hyperfocal problem. Requiring the residual to stay under $\tau$ pixels puts $1/z$ inside a band of width $2\tau/fb$ about $1/z_0$, which in depth is

$$\left[\frac{1}{1/z_0 + \tau/fb}, \ \frac{1}{1/z_0 - \tau/fb}\right],$$

a near limit and a far limit that runs to infinity as soon as $z_0 \ge fb/\tau$. That threshold is a hyperfocal distance in the exact sense: register at $H = fb/\tau$ and everything from $H/2$ to infinity is inside tolerance, which is the same $H$-to-$H/2$ relation a lens focused at its hyperfocal distance gives. For the rig above, $H = 76$ metres and the zone is 38 metres to the horizon.

The correspondence is not an analogy. Depth of field is a tolerance on a blur circle that grows with $|1/z - 1/z_{\text{focus}}|$, and a stitch residual is a tolerance on a disparity that grows with $|1/z - 1/z_0|$; both are linear in inverse depth with a constant made of a focal length and a small length — an aperture in one case, a baseline in the other. Every rule photographers have for depth of field transfers unchanged, including the one that matters most here: the near limit is where the trouble is, and focusing further away buys the far end cheaply and the near end not at all.

Why the harmonic mean, in words #

The algebra is three lines and the reason is one sentence, and the sentence is worth having because it transfers.

The stitch’s error is linear in $1/z$, not in $z$. So the natural coordinate of the problem is inverse depth, and every question about “the middle of the range” has to be asked there. The middle of $[1/z_f, 1/z_n]$ is $\frac12(1/z_n + 1/z_f)$, and the depth whose reciprocal that is, is the harmonic mean. Nothing about harmony is involved; it is the arithmetic mean, taken in the variable the problem is linear in.

This site meets the same move in three other places and it is the same move each time. The depth a stereo pair reports is a reciprocal, so a symmetric error in the picture is an asymmetric interval in the world. A depth buffer stores a reciprocal, so its precision is spent near the eye. A frustum’s near plane can be moved almost freely at the far end and not at the near one.

The transferable form is short: find the variable the error is linear in, and do the averaging there. Averaging in metres a quantity that lives in reciprocal metres is how the arithmetic midpoint gets chosen, and the factor of two is what it costs.

Checked against pictures rather than against itself #

Everything above is algebra about a rig, and this site’s habit is that algebra about a thing is not a measurement of it.

So the closed form is checked against two actual cameras. rigPair builds a rectified pair at a stated baseline, projects six points at depths from two metres to twenty-four, applies the shift a warp for $z_0$ would apply, and asks where each point landed. The residuals agree with $f b (1/z - 1/z_0)$ over the whole set, and the point at the stitching depth itself registers exactly, which is the control.

That control is the one that matters. A closed form and an implementation can share a sign error and agree perfectly; a point at the depth the warp was built for has to land at zero, and if it does not, the whole comparison is between two versions of the same mistake.

Why this is not the pivot error with a bigger number #

It looks like the same problem at a larger scale and it is not, and the difference is what a reader can do about it.

A pivot error is a mounting mistake with a remedy: slide the camera back on a rail until the pupil sits on the axis and both halves of the parallax go to exactly zero. The whole of the previous rung is about how much of it can be bought off without the rail, and the answer is a specification rather than a fix.

A rig’s baseline is not a mistake and has no remedy. Two lenses cannot be in the same place, so the residual has a floor set by the hardware’s size, and the only choices available are which surface to be right on and how big to build the rig.

There is also a difference in what the residual is a residual of. The pivot error’s miss is a length in the room, computable before any scene exists, because it is a property of the rays. A rig’s residual is a disparity in pixels and needs a depth before it is a number at all — the geometry is fixed and the scene supplies the divisor.

What the rig gets in exchange #

It is worth stating what the baseline buys, because an essay that only prices a defect reads as an argument against the thing.

A rig of several cameras exposes every direction at the same instant. No sweep, no sequence of frames seconds apart, so the tear that a stitched panorama puts through a moving subject does not happen — a person walking through a rig’s picture is drawn once, in the right shape, from one of the several eyes.

And it needs no moving parts and no time. A rig can record video; a swing-lens camera and a tripod sequence cannot record a moment.

So the three arrangements this field now carries divide up cleanly. The swing lens has one centre and pays in detail and in a shear. The tripod sequence has many centres a rail can collapse to one, and pays in time. The rig has many centres nothing can collapse, and pays in a floor — and buys the only thing the other two cannot have, which is simultaneity.

What a rig would have to know to do better #

The residual is a function of depth, so a warp that knew the depth of every pixel would remove it exactly. That is not a rhetorical observation — it is what a rig with a depth sensor does, and it works.

What it costs is the premise. A stitch that consults a depth map is no longer a projection at all: different parts of the picture are warped by different amounts, so there is no single map from directions to marks, and every quantity this field computes about a picture surface stops applying. The result is a rendering rather than a photograph, and the distinction is the one the pipeline field draws between a projection and a program that produces a picture.

So the rig has three settings and they are not on a scale. Register on the far field and accept the near disparity; register on a chosen depth and accept half the spread; or measure the depth and give up being a projection. This site can price the first two exactly and can only say what the third gives up.

The surface the picture is right on #

There is a way of reading the whole result that puts it in this field rather than in depth, and it is the reason the essay sits where it does.

A rig’s stitch does not produce a picture with an error in it. It produces a picture that is a correct projection of one surface — the sphere of radius $z_0$ — with everything else drawn as though it were on that surface. The doubling at a near object is not noise; it is the picture faithfully reporting two directions for a thing that is not where the picture assumes it is.

That is the same move the second-eye essay makes on a two-centre drawing: instead of asking how wrong the picture is, ask what arrangement it would be a correct picture of. Here the answer is a shell at one distance, which is a stranger scene than a sheared room and is just as much a scene.

What this does not settle #

It does not say which depth a real stitcher should use. Real scenes are not intervals with two ends; they have a distribution of depths, and minimising the worst case over a range is a different objective from minimising what a viewer notices.

It does not model the seam’s placement. A stitcher chooses where to cut as well as what to warp, and cutting through a region with nothing in it removes the artefact without removing the disparity — which is a good practical answer and is not a geometric one, in the same way choosing where to put the horizon is a choice about the drawing rather than about the room.

It does not treat the cameras’ own errors. Each of them has a lens, and a lens bends straight lines by an amount that is usually larger than any number here; a rig’s calibration is a bigger problem than its baseline and a different one.

And it does not say a rig is worse than a rail. It says the rig’s residual has a floor and the rail’s does not, which is a statement about the arrangements rather than about the pictures they make.

When an error cannot be removed, the useful question is which one thing it can be made zero at. The answer is a surface, and stating that surface is more informative than stating the error — because the surface is a choice and the error is a consequence.