Visual Plane
every picture states the point it is correct from
Surfaces that are not flat

A rig is right on one surface

Several cameras looking outward have several centres, and no warp registers all of a scene. The residual is a disparity, so the depth the stitch is computed for has an exact optimum — the harmonic midpoint of the depth range, not the middle of it — and the arithmetic middle costs a factor of 2 z_far over the sum, which tends to two.
23 min read 15 figures Not a projection at allDetermined up to somethingFitted, not assumed

Worth reading first: The pivot that is not the eye · Depth is a reciprocal · What a 360-degree photograph actually is.

The two rungs before this one are about a mistake: a camera bolted to its tripod by the wrong point, when the right point was available and a rail would have found it. The rig is not a mistake. Put several cameras on a ball and point them outward and they have several entrance pupils, several centimetres apart, and nothing can be done about it — two lenses cannot occupy the same place.

So the question changes. It is no longer how badly is this mounted but which depth should the stitch be right at, since it cannot be right at all of them.

A rig's stitch is right on one surface and nowhere elseThe residual a rig's warp leaves, against depth, for three choices of the depth it was computed for: the harmonic midpoint at 2.86 m, the arithmetic one at 15.8 m, and one close in at 1.65 m. Each curve crosses zero exactly once, at its own stitching depth, and the crossing is the surface the picture is correct on. Everywhere else the same feature is drawn twice, or cut in half, by f·b·(1/z − 1/z₀) pixels — a disparity, so it is reciprocal in depth and the near end always costs more than the far one. There is no fourth curve that is flat.-500500.5001distance to the thing being stitched (log₁₀ metres)what the stitch leaves behind (px)2.86 m80 mm baselinezero at one depth each
Fig. 1 What a rig’s warp leaves behind, against depth, for three choices of the depth it was computed for. Each curve crosses zero exactly once, and the crossing is the surface the picture is correct on.

The residual is a disparity, and disparities are reciprocal

Two cameras a baseline bb apart, both looking the same way, image a point at depth zz at positions differing by fb/zf b / z pixels. That is the stereo disparity the depth field is built on, and it is what a rig’s stitch is fighting.

A warp computed to register depth z0z_0 is a shift of fb/z0f b / z_0. Apply it and what is left over is

Δ(z)  =  f b(1z−1z0)\Delta(z) \;=\; f\,b\left(\frac1z - \frac1{z_0}\right)

which is zero at z0z_0 and nowhere else. The picture is correct on one surface — the sphere or cylinder of radius z0z_0 around the rig — and every feature off it is drawn twice, or cut in half where the seam runs through it.

A rig's stitch is right on one surface and nowhere elseThe residual a rig's warp leaves, against depth, for three choices of the depth it was computed for: the harmonic midpoint at 2.86 m, the arithmetic one at 15.8 m, and one close in at 1.65 m. Each curve crosses zero exactly once, at its own stitching depth, and the crossing is the surface the picture is correct on. Everywhere else the same feature is drawn twice, or cut in half, by f·b·(1/z − 1/z₀) pixels — a disparity, so it is reciprocal in depth and the near end always costs more than the far one. There is no fourth curve that is flat.-10001000.5001distance to the thing being stitched (log₁₀ metres)what the stitch leaves behind (px)2.86 m160 mm baselinezero at one depth each
Fig. 2 A wider baseline. The shape of the curves is unchanged and the vertical scale is not: the residual is linear in the baseline and reciprocal in the depth, so doubling the rig’s size doubles every number on the plot.

The reciprocal is doing the work in everything that follows. A depth range from a metre and a half to thirty metres is a factor of twenty in distance and a factor of twenty in disparity, and the near end dominates so heavily that any choice made by looking at the metres will be wrong.

The best depth is the harmonic midpoint

Ask for the choice of z0z_0 that makes the worst residual over a range [zn,zf][z_n, z_f] as small as possible. The residual is monotone in 1/z1/z, so the worst is always at one end or the other, and the minimum of the worse of the two is where they are equal:

1zn−1z0  =  1z0−1zf⟹z0  =  2znzfzn+zf\frac1{z_n} - \frac1{z_0} \;=\; \frac1{z_0} - \frac1{z_f} \qquad\Longrightarrow\qquad z_0 \;=\; \frac{2 z_n z_f}{z_n + z_f}

which is the harmonic mean of the two ends. For a scene from one and a half metres to thirty, that is under three metres — a fifth of the way along the range in metres, and exactly halfway along it in disparity.

The best stitching depth is the harmonic middle, not the middleThe worst misregistration anywhere between 1.5 m and 30 m, against the depth the rig's warp was computed for, at a 80 mm baseline. The minimum is at 2.86 m — the harmonic midpoint 2 z_n z_f/(z_n + z_f) — where the worst case is 32.9 px. The arithmetic middle, 15.8 m, is the choice anybody would make and costs 62.7 px, worse by exactly 2 z_f/(z_n + z_f) = 1.905. And the minimum is 32.9 px rather than nothing: it is half the disparity spread of the range, so a rig with a baseline registers one surface and nothing else.02040600.5001the depth the stitch was computed for (log₁₀ metres)the worst residual in the scene (px)2.86 m15.8 m80 mm baseline · 1.5–30 m32.9 px at best
Fig. 3 The worst residual anywhere in the scene, against the depth the warp was computed for. The minimum is at the harmonic midpoint, and the middle of the range — the choice anybody would make — sits well up the right-hand side of the valley.

The search in that figure is not decoration. The closed form above is three lines of algebra and three lines of algebra can be wrong; a sweep over four thousand candidate depths that lands on the same number is what makes it a measurement. assertTheStitchingDepthIsTheHarmonicMidpoint requires the search’s minimum to agree with the formula to two parts in a thousand, which is the resolution of the sweep rather than of the claim.

The penalty for the obvious choice, in closed form

The arithmetic midpoint is what a person picks. It is wrong, and by how much has an exact answer:

worst at the arithmetic midpointworst at the harmonic one  =  2zfzn+zf\frac{\text{worst at the arithmetic midpoint}}{\text{worst at the harmonic one}} \;=\; \frac{2 z_f}{z_n + z_f}

For a range of one and a half to thirty metres that is a shade over one point nine — nearly twice as bad. And as the far end runs away it tends to exactly two and never exceeds it, so the penalty for the natural choice is bounded, and the bound is a factor of two.

The best stitching depth is the harmonic middle, not the middleThe worst misregistration anywhere between 1.5 m and 120 m, against the depth the rig's warp was computed for, at a 80 mm baseline. The minimum is at 2.96 m — the harmonic midpoint 2 z_n z_f/(z_n + z_f) — where the worst case is 34.2 px. The arithmetic middle, 60.8 m, is the choice anybody would make and costs 67.6 px, worse by exactly 2 z_f/(z_n + z_f) = 1.975. And the minimum is 34.2 px rather than nothing: it is half the disparity spread of the range, so a rig with a baseline registers one surface and nothing else.02040600.50011.502the depth the stitch was computed for (log₁₀ metres)the worst residual in the scene (px)2.96 m60.8 m80 mm baseline · 1.5–120 m34.2 px at best
Fig. 4 A deeper scene. The valley’s floor rises and the arithmetic midpoint moves further up the side of it, toward the factor of two the closed form says it approaches and never passes.

That bound is the useful half. A rig whose stitching depth was set by somebody eyeballing the middle of the room is not catastrophically wrong; it is within a factor of two of the best it could have been, forever, whatever the scene. Which is worth knowing because it says where not to spend effort: the depth is worth choosing well and it is not worth choosing perfectly, and the thing that actually matters is the next paragraph.

And the floor, which is why the choice is small

Put the harmonic midpoint back into the residual and the worst case comes out as

12 f b(1zn−1zf)\tfrac12\, f\, b \left(\frac1{z_n} - \frac1{z_f}\right)

— half the disparity spread of the whole depth range. That is the best any choice of z0z_0 can achieve, and it is not zero. On an eighty-millimetre baseline over a range from one and a half to thirty metres it is tens of pixels, which is a visible doubling.

A rig's stitch is right on one surface and nowhere elseThe residual a rig's warp leaves, against depth, for three choices of the depth it was computed for: the harmonic midpoint at 2.86 m, the arithmetic one at 15.8 m, and one close in at 1.65 m. Each curve crosses zero exactly once, at its own stitching depth, and the crossing is the surface the picture is correct on. Everywhere else the same feature is drawn twice, or cut in half, by f·b·(1/z − 1/z₀) pixels — a disparity, so it is reciprocal in depth and the near end always costs more than the far one. There is no fourth curve that is flat.-100100.5001distance to the thing being stitched (log₁₀ metres)what the stitch leaves behind (px)2.86 m20 mm baselinezero at one depth each
Fig. 5 A small rig. The residual is linear in the baseline, so the only complete repair is a baseline of nothing — which is the control, and is a rig with one camera in it.

So the optimisation is real and it is not a repair. assertNoStitchingDepthIsRight exists to stop the previous section being read as one: it requires the best achievable worst case to equal the closed form for the floor, and requires that floor to be pixels rather than nothing, and then requires it to vanish exactly when the baseline does. Without the third the floor could be an artefact of the fit; without the second the whole essay could be read as a tuning guide.

Depth from disparity, with the 1 px the reading is worthZ = fB/d on a 90 mm baseline at 900 px. The line is exact — it returns the camera's own depth to 8e-13 m. The band is what 1 px of disparity error costs, and it stops being a ±. At 6.5 m it runs 6.03–7.08 m, lopsided by 1.17, and the textbook ±Z²δ/fB is 0.6% out. At 40 m it runs 26.8–79.0 m — 39.0 m beyond the estimate against 13.2 m before it, a lopsidedness of 2.95 — and the same formula is 24% out. Past 81.0 m the far edge is infinity.02040608010203040true depth (m)depth reported from the disparity, with a 1 px reading error6.03–7.08 m14.95–23.69 m26.78–79.02 mat 40 m: +39.0 m against −13.2 munbounded past 81.0 m
Fig. 6 The reciprocal itself, from the depth field. Everything in this essay is a consequence of the fact that a fixed error in the picture maps to an interval in depth that is not symmetric and eventually is not bounded.

Why the harmonic mean, in words

The algebra is three lines and the reason is one sentence, and the sentence is worth having because it transfers.

The stitch’s error is linear in 1/z1/z, not in zz. So the natural coordinate of the problem is inverse depth, and every question about “the middle of the range” has to be asked there. The middle of [1/zf,1/zn][1/z_f, 1/z_n] is 12(1/zn+1/zf)\frac12(1/z_n + 1/z_f), and the depth whose reciprocal that is, is the harmonic mean. Nothing about harmony is involved; it is the arithmetic mean, taken in the variable the problem is linear in.

This site meets the same move in three other places and it is the same move each time. The depth a stereo pair reports is a reciprocal, so a symmetric error in the picture is an asymmetric interval in the world. A depth buffer stores a reciprocal, so its precision is spent near the eye. A frustum’s near plane can be moved almost freely at the far end and not at the near one.

Where a depth buffer's codes actually goThe share of a 24-bit buffer used up within each distance, for a near plane at 0.1 m and a far plane at 1000 m. Ninety per cent of the range is spent inside the first metre; the last nine hundred metres share what is left.near 0.1 m · far 1000 m · 24-bit codeswithin 20 cm50.0%within 50 cm80.0%within 1 m90.0%within 5 m98.0%within 50 m99.8%half the codes are gone by 0.20 m — the harmonic mean of the two planesshare of the depth buffer used up within each distancearithmetic midpoint would be 500 m
Fig. 7 The same reciprocal, spending a buffer’s precision. A quantity stored as 1/z has its resolution where the reciprocal changes fastest, which is exactly where a stitching error is worst.

The transferable form is short: find the variable the error is linear in, and do the averaging there. Averaging in metres a quantity that lives in reciprocal metres is how the arithmetic midpoint gets chosen, and the factor of two is what it costs.

Checked against pictures rather than against itself

Everything above is algebra about a rig, and this site’s habit is that algebra about a thing is not a measurement of it.

So the closed form is checked against two actual cameras. rigPair builds a rectified pair at a stated baseline, projects six points at depths from two metres to twenty-four, applies the shift a warp for z0z_0 would apply, and asks where each point landed. The residuals agree with fb(1/z−1/z0)f b (1/z - 1/z_0) over the whole set, and the point at the stitching depth itself registers exactly, which is the control.

That control is the one that matters. A closed form and an implementation can share a sign error and agree perfectly; a point at the depth the warp was built for has to land at zero, and if it does not, the whole comparison is between two versions of the same mistake.

Disparity to depth, and the wall the reader's own head puts inz = b·D/(b − d). Zero disparity puts the point on the screen at 4.0 m; crossed disparity brings it forward; and at d = 63 mm — the separation of the eyes — the point reaches infinity. Past that the display is asking the eyes to diverge, which they cannot do, so the depth budget is set by the width of the reader's head and by nothing about the scene.010203040-2502550disparity on the screen — millimetreswhere the point is depicted — metres from the eyesd = 63 mm — the eyes' separationon the glassscreen at 4.0 m · eyes 63 mm apartthe ceiling is the head, and it does not move when the screen does
Fig. 8 The same arithmetic in the screen field, where the two pictures are shown rather than stitched. A disparity is a disparity whether it is being removed or displayed.

Why this is not the pivot error with a bigger number

It looks like the same problem at a larger scale and it is not, and the difference is what a reader can do about it.

A pivot error is a mounting mistake with a remedy: slide the camera back on a rail until the pupil sits on the axis and both halves of the parallax go to exactly zero. The whole of the previous rung is about how much of it can be bought off without the rail, and the answer is a specification rather than a fix.

A rig’s baseline is not a mistake and has no remedy. Two lenses cannot be in the same place, so the residual has a floor set by the hardware’s size, and the only choices available are which surface to be right on and how big to build the rig.

A panorama pivoted 60 mm behind its own pupilLooking down on 6 frames taken by turning a camera about a point 60 mm from where the light actually crosses. The small circle is the path the entrance pupil takes; the heavy rays are each frame's own axis and they pass through the pivot exactly; the lighter rays are the edges of the strip each frame contributes, and they miss it by e·sin γ — 30.0 mm at 30.0° off axis. Every ray the stitch uses is tangent to a circle of that radius, drawn here, so the picture has a radius where a projection would have a point. The rays of the whole strip, top of frame to bottom, miss their own least-squares centre by 24.98 mm.the pivottangent circle, 30.0 mmno single viewpoint — the rays miss by 24.98 mm6 frames · pivot 60 mm off
Fig. 9 The other arrangement, where the offset is an error rather than a dimension. Its rays are tangent to a sphere whose radius the photographer chose by accident; a rig’s are separated by a baseline somebody chose on purpose.

There is also a difference in what the residual is a residual of. The pivot error’s miss is a length in the room, computable before any scene exists, because it is a property of the rays. A rig’s residual is a disparity in pixels and needs a depth before it is a number at all — the geometry is fixed and the scene supplies the divisor.

Two rays, 5.07 mm apart, in the plane that contains bothThe ray from the left eye through its mark and the ray from the right eye through its. With the marks placed exactly they meet, to 1.9e-15 m. With the same marks read to 1 px they miss by 5.07 mm at a range of 7.19 m. Triangulation is not an intersection; the reported point is a choice about what to minimise, and the gap is the part a residual alone will not tell you.midpoint — 5.07 mm gapfrom the left eyefrom the right eyegap 5.07 mm at 7.19 mexact marks: 1.9e-15 m
Fig. 10 The general case underneath both: two rays from two places, and the point they do not meet at. Everything in this field is a way of measuring that miss and deciding what to do about it.

What the rig gets in exchange

It is worth stating what the baseline buys, because an essay that only prices a defect reads as an argument against the thing.

A rig of several cameras exposes every direction at the same instant. No sweep, no sequence of frames seconds apart, so the tear that a stitched panorama puts through a moving subject does not happen — a person walking through a rig’s picture is drawn once, in the right shape, from one of the several eyes.

And it needs no moving parts and no time. A rig can record video; a swing-lens camera and a tripod sequence cannot record a moment.

So the three arrangements this field now carries divide up cleanly. The swing lens has one centre and pays in detail and in a shear. The tripod sequence has many centres a rail can collapse to one, and pays in time. The rig has many centres nothing can collapse, and pays in a floor — and buys the only thing the other two cannot have, which is simultaneity.

Neither instrument is exact, and they are inexact in different thingsSix numbers on one arrangement: a 6-frame stitch with the pivot 60 mm off the pupil against a swing-lens camera, both delivering the same panorama, with a subject 1.4 m/s at 3 m. The swing lens has no parallax — exactly none, because it turns about its own pupil — and pays for it twice: 2.08 times less detail round the turn, and a 32 px shear on anything that moves, because its two ends are exposed a moment apart. The stitched camera's own motion failure is 956 px and is a different shape: a tear at the seam rather than a bend, because its frames are seconds apart rather than continuous. A shear is a picture of a distorted subject; a tear is a picture of no subject.stitched · parallax at the seam26swing lens · parallax0stitched · detail, px per degree100swing lens · detail, px per degree48stitched · a moving subject, torn1410swing lens · a moving subject, sheared686 frames · pivot 60 mm offdetail ratio 2.08×
Fig. 11 Two of the three, priced on one arrangement. The third’s column would have a zero in the motion row and a number in the parallax row that no adjustment removes.

What a rig would have to know to do better

The residual is a function of depth, so a warp that knew the depth of every pixel would remove it exactly. That is not a rhetorical observation — it is what a rig with a depth sensor does, and it works.

What it costs is the premise. A stitch that consults a depth map is no longer a projection at all: different parts of the picture are warped by different amounts, so there is no single map from directions to marks, and every quantity this field computes about a picture surface stops applying. The result is a rendering rather than a photograph, and the distinction is the one the pipeline field draws between a projection and a program that produces a picture.

So the rig has three settings and they are not on a scale. Register on the far field and accept the near disparity; register on a chosen depth and accept half the spread; or measure the depth and give up being a projection. This site can price the first two exactly and can only say what the third gives up.

The surface the picture is right on

There is a way of reading the whole result that puts it in this field rather than in depth, and it is the reason the essay sits where it does.

A rig’s stitch does not produce a picture with an error in it. It produces a picture that is a correct projection of one surface — the sphere of radius z0z_0 — with everything else drawn as though it were on that surface. The doubling at a near object is not noise; it is the picture faithfully reporting two directions for a thing that is not where the picture assumes it is.

That is the same move the second-eye essay makes on a two-centre drawing: instead of asking how wrong the picture is, ask what arrangement it would be a correct picture of. Here the answer is a shell at one distance, which is a stranger scene than a sheared room and is just as much a scene.

The same picture, from one eye, of a different roomThe thick outlines are the far boxes as the second eye drew them. The thin outlines are the first eye's picture of a moved copy of those boxes — moved by one projective map of space, the one that holds the picture plane still point by point and carries the second eye onto the first. The two lie on top of each other to 4.9e-13 px, over 16 corners. Without the map the same first eye would have drawn those boxes 901 px away. So the two-eyed picture is a one-eyed picture, of a room that is not the room.apart by 4.9e-13 pxcorrect from 17 cm, at 160 mm wideone eye again
Fig. 12 The move this is an instance of: a picture no single camera makes, read as a correct picture of something else. A rig’s stitch is a correct picture of a scene painted on a sphere.
The same 110° view, projected onto a plane and onto a cylinderOn the plane every straight line stays straight (1e-13 px of bend) and the edges stretch; on the cylinder the stretch is even and straight lines bow by up to 544 px.flat picture plane — straight lines stay straightcylindrical picture surface — even stretch, bowed linesone scene, two picture surfacesneither is the distorted one
Fig. 13 And the field’s own premise, which the reading depends on: the surface a picture is on is a choice, and a surface at a stated distance is as much a choice as a surface of a stated shape.

What this does not settle

It does not say which depth a real stitcher should use. Real scenes are not intervals with two ends; they have a distribution of depths, and minimising the worst case over a range is a different objective from minimising what a viewer notices.

It does not model the seam’s placement. A stitcher chooses where to cut as well as what to warp, and cutting through a region with nothing in it removes the artefact without removing the disparity — which is a good practical answer and is not a geometric one, in the same way choosing where to put the horizon is a choice about the drawing rather than about the room.

It does not treat the cameras’ own errors. Each of them has a lens, and a lens bends straight lines by an amount that is usually larger than any number here; a rig’s calibration is a bigger problem than its baseline and a different one.

And it does not say a rig is worse than a rail. It says the rig’s residual has a floor and the rail’s does not, which is a statement about the arrangements rather than about the pictures they make.

When an error cannot be removed, the useful question is which one thing it can be made zero at. The answer is a surface, and stating that surface is more informative than stating the error — because the surface is a choice and the error is a consequence.

One half falls away and the other does not moveThe worst miss across the seam and the worst miss at the top of the frame, against the number of frames, for a pivot 60 mm off the pupil and a frame 38° tall. The first is e·sin(π/n) and falls from 60.0 mm to 2.9 mm; the second is e·sin(β) and is 19.53 mm at every count, bit for bit. They cross at π/β = 9.47, so past 10 frames every remaining pixel of parallax is vertical and no further shooting touches it. The upper curve is the corner of the strip, which is what a reader actually gets.0204060204060frames in the panoramadistance from the pivot (mm)π/β = 9.5across the seamup the framepivot 60 mm · frame 38° tallfloor 19.53 mm
Fig. 14 The neighbouring arrangement’s two halves, for comparison. A pivot error’s horizontal half can be shot away and a rig’s baseline cannot be anything away, which is the difference between a mistake and a dimension.
Where a stereo pair stops being able to say "no further than"The disparity of a point at range Z is fB/Z, so once it falls below the 1 px the reading is worth, the far edge of the depth interval is infinity. That happens at fB/δ, and for a human pair of eyes at 1062 px it is 69.0 m. Doubling the baseline doubles it; nothing about the estimator does.123-2-10baseline (m, log scale)range past which 1 px of disparity error is unbounded (m, log scale)65 mm → 69.0 mrange = fB/δ — 69.0 m at a 65 mm baseline1062 px focal length, 1 px reading
Fig. 15 The other consequence of the reciprocal, from the depth field: the distance past which a pair of views stops distinguishing depths at all.

Where this goes next

the pivottangent circle, 30.0 mmno single viewpoint — the rays miss by 24.98 mm6 frames · pivot 60 mm off The pivot that is not the eye A camera bolted to a tripod turns about the screw under its baseplate, and the light crosses somewhere else. Every ray of the panorama that results passes the pivot by e sin γ — the offset times the sine of how far off its own frame's axis the ray points — so the picture has a radius where a projection would have a point. 025507510010203040true depth (m)depth reported from the disparity, with a 1 px reading error5.86–7.33 m13.96–26.69 m23.76–126.49 mat 40 m: +86.5 m against −16.2 munbounded past 58.5 m Depth is a reciprocal Two eyes measure a shift in the picture, and depth is that shift divided into a constant. So a fixed error in what is read maps to an interval in what is reported that is not centred on the answer, and at forty metres runs sixteen metres nearer and eighty-six further. no picture at 300°the plane is unbounded at 180°plane — bend 0.0%cylinder — bend 6.0%stereographic — bend 3.5%equidistant — bend 4.5%equal-area — bend 5.1%equirect. — bend 5.7%300° across in every panelsame scene, same angle, six surfaces What a 360-degree photograph actually is The format every spherical camera writes preserves nothing — not straightness, not shape, not area — and it is the right choice anyway, for a reason that has nothing to do with looking at it. An equirectangular file is a lookup table of directions, and the picture only exists at the moment something re-projects a piece of it.

What links here

Computed from the collection, not written here: the essays that point at this one.

Shares its objects with

Essays that name at least two of the same things, and that neither author linked.

Named objects

A flat tag is an object no other essay names yet.

Baselinecentre of projectionDemonstrationDepth from disparityDisparityMinimaxPanoramaParallaxPicture surfaceStitching