A sway gives the blind centre a depth, not a good one

An epipole in the picture leaves a blind disc followed a camera that steps straight forward between two pictures. The image of each camera sits in the middle of the other’s picture, every mark slides straight away from that point between the pictures, and a mark at the point itself does not slide at all, whatever its depth. Around it lies a disc inside which a pixel of reading error swamps the slide — 122.7 px in radius for a surface 8 m away, with no depth recovered at its centre at any range.

That is the case of every camera on a vehicle heading somewhere, and a vehicle does not drive in a perfectly straight line. It sways. The essay closed by asking whether that sway is worth anything: whether a third picture taken slightly off the line gives the marks near the centre a second slide, across rather than along, and how large the sway has to be before the blind disc closes.

The answer is that any sway at all closes the hole at the centre, and that no sway a steering wheel produces closes the disc.

The measurement #

Three pictures, as a camera driving forward would take them: the first at the start, the second half a metre on, the third a metre on, all looking straight ahead with a 60° field on a picture 690 px wide. The middle one is displaced sideways by a stated sway. For each mark in the first picture, the point on its ray at a stated distance is projected into all three pictures, and the depth error a reading error of one pixel in every coordinate of every picture leaves on that point is computed from the least-squares intersection of the three rays. The error is quoted as a share of the depth, as the earlier essay quoted it.

This is a slightly different measure from the earlier essay’s, which moved one mark by one pixel along its epipolar line in the second picture. Here every picture is read with the same error in every direction, which is the fair measure once there are three pictures and the lines they constrain no longer agree. The two-picture disc on this measure is 206 px in radius at 8 m.

Any sway closes the hole #

The hero figure draws the curve inside which a pixel costs a tenth of the depth of a surface 8 m away, for three arrangements. Two pictures: a disc 206 px in radius, with no depth at all at its centre. Three pictures in a line: the longer forward baseline shrinks the disc to 92 px, and the centre is still blind, since three cameras on one line all see the mark at the epipole in one direction. Three pictures with the middle one swayed 5 cm sideways: 83 px, and the centre now has a depth.

The centre has a depth because the swayed picture sees it from off the line. The three rays through the centre mark are no longer one line, so they meet at a point, and the reading error moves that point by a finite amount. The hole is closed. What is left is how finite.

At the centre, and at 8 m, a pixel of reading costs 288 per cent of the depth for a sway of half a centimetre, 144 per cent for a centimetre, 72 for two, 29 for five, 14 for ten and 3 for fifty. The cost is one constant times the depth over the sway, to within five per cent across the whole range and at 4, 8 and 16 m: $1.80\times10^{-3}\,Z/s$. The constant is close to one over the focal length in pixels, $1.67\times10^{-3}$, which says what the centre’s depth is resting on. It is resting on the sway alone, as if the sway were the baseline of a sideways pair — depth is a reciprocal gave such a pair’s depth error as the depth squared over the focal length times the baseline, per pixel, and divided by the depth that is exactly the form measured. The forward step contributes nothing at the centre, because along the line of travel the centre mark does not slide.

A tenth of the depth at the centre needs a sway of 7 cm at 4 m, 14 cm at 8 m and 29 cm at 16 m, and beyond about 28 m it needs a sway larger than the 50 cm forward step itself. A lateral wobble of a centimetre or two between frames half a metre apart — the scale of a small steering correction, if a vehicle’s is that size — buys the centre mark a depth good to between 70 and 150 per cent at 8 m: a number, where before there was none, but not a number anything should be steered by.

The disc shrinks slowly #

The centre was only the worst point. The disc around it, where a pixel costs more than a tenth of the depth, is the part of the picture that matters to anything looking ahead.

With no sway the disc is 92 px in radius. A centimetre of sway leaves it at 91 px across the picture and 92 up it; 5 cm, at 83 and 87 px; 10 cm, at 59 and 66. It closes altogether only at a sway of 15 cm, where the centre itself reaches a tenth of the depth. The sway helps across the picture slightly more than up it, since a sideways baseline slides marks sideways, and it helps the disc’s edge least, because at the edge the forward step’s own slide is already doing most of the work.

So the sway does two different things at two different places. At the centre it is everything — the only source of depth there is — and its value is the sway over the depth. At the edge of the disc it is a small addition to a slide that already exists, and it moves the edge by a few pixels for a sway of a few centimetres. Closing the disc needs a sway of 15 cm, nearly a third of the forward step, at 8 m, and proportionally more for anything further away.

The floor under the cost #

The profile from the centre outward shows the two regimes meeting.

With no sway, the cost rises without limit toward the centre: 1,848 per cent half a pixel out, and more closer in. With a sway it levels off at the value the sway alone gives — 72 per cent for 2 cm, 14 for 10 cm, 6 for 25 cm — and the level is reached where the forward step’s slide falls below the sway’s. Far from the centre the curves merge: beyond about 64 px a 2 cm sway changes the cost by less than five per cent, because the forward step’s slide there is already larger than anything a sway adds.

That is the shape a third ray is worth what its picture is worth would predict from the other direction. A ray adds information in proportion to the angle it makes with the rays already there, and at the centre the forward pictures’ rays make no angle with each other at all, so the swayed ray is the only one that counts. Away from the centre the forward pictures’ rays already cross at an angle, and the swayed ray is one more ray among them.

Which picture sways matters off the centre, not at it #

A real sway is not confined to one picture. The last measurement moves the sway around the sequence, with the same 5 cm in each case.

At the centre it barely matters which picture sways: 28.8 per cent with the middle one swayed, 26.0 with the last and 28.6 with the first. The centre’s depth rests on the sway alone, so the sway’s size and the distance to the point decide it, not the sway’s place in the sequence. A fourth picture taken in line adds nothing at the centre however far on it is taken; a fourth picture with the third swayed brings the centre to 23.9 per cent.

Off the centre the choice matters, and in a direction worth knowing. A swayed picture has its own epipole, displaced from the centre of the first picture by the focal length times the sway over its forward distance — 30 px for a 5 cm sway on the last picture, a metre on. Near that point the swayed picture’s own slide vanishes, exactly as the forward pictures’ does at the centre, and a mark there loses what the swayed picture would have contributed. At 40 px aside, toward the swayed picture’s epipole, the cost is 51.4 per cent with the last picture swayed against 23.1 with all three in line. A sway does not remove the blind point; it adds a second, smaller one beside it, and lets the two share the blindness.

What a closed disc costs #

It is worth seeing the arrangement in which the disc does close, because it shows what the forward-driving camera would have to become.

With the middle picture swayed 25 cm — half the forward step — there is no disc at all at 8 m. The centre’s depth is good to 6 per cent a pixel and every other mark in the picture does better than a tenth. But the camera that took these three pictures was not driving forward in any useful sense. Its middle picture sat a quarter of a metre off the line, the width of a lane marking, and its track between three frames was a zig-zag as wide as it was long. That is not a vehicle swaying; it is a vehicle whose camera has been deliberately moved sideways as much as forward, which is to say a stereo arrangement with a forward component, not a forward one with a wobble.

The two-picture disc of the earlier essay and this closed one are the ends of one scale, and the scale is the ratio of the sideways baseline to the forward one. The earlier essay’s comparison of a forward step with a sideways step of the same size is the far end; the sway is the near end; and nothing on the scale between them lets a camera look straight ahead, drive straight on and also see how far away the thing ahead is.

A third picture is a third ray, and the rays decide #

The earlier readings with three pictures in this subject point the same way from other directions. A third eye that lands on the next post found a third picture exposing a wrong match by hundreds of pixels at most places and confirming it at a few: a third view is worth exactly what its position lets it see. The centre mark here is the limiting case. A third picture on the same line as the first two sees the centre along the same ray again, and adds nothing however far along the line it is taken.

The spread a point gets measured, for a camera moving round a scene, that each point’s error follows the angle its rays subtend at it rather than the arc the track covers. At the centre of a forward-driving camera’s picture that angle is zero for every picture taken on the line, and the sway’s whole contribution is the one small angle it opens, $s/Z$ in radians, which is why the cost goes as the depth over the sway. The quantisation whole pixels cut space into shells found for a sideways pair applies to that small angle too: a matcher reading whole pixels divides the centre’s depth into shells a pixel of slide apart, and for a 5 cm sway at 8 m a pixel of slide is 29 per cent of the depth, so the centre’s depth can only ever read as one of a handful of values.

Why a vehicle cannot sway its way out #

Everything above is a statement about geometry, and it turns into a statement about vehicles with one more number.

The disc closes when the sway, as a sideways baseline, gives the centre the precision the forward step gives the disc’s edge. At the centre the cost is $Z/(f s)$ per pixel, near enough, so a tenth of the depth needs $s \approx 10\,Z/f$: 14 cm at 8 m with this camera, 1.4 m at 80 m. A vehicle that swayed by that much between frames would be a vehicle out of control, and a camera fixed to it would spend most of its pictures on the road’s edges. The practical conclusion is the unwelcome one: the blind disc of a forward-driving camera is a property of driving forward, and wobble is not a remedy for it.

What the sway does buy is qualitative. A depth with an honest error bar of 150 per cent is still a depth — it says the thing ahead is not at infinity, and it gets better as the thing gets nearer, since the cost scales with the depth. The image of the other eye found that a pair whose epipole is in the picture has a whole class of marks it can say nothing about; a sequence with any sway at all has none, only marks it can say little about. That distinction matters to anything that has to decide whether a depth exists before deciding what it is.

What the measurement leaves alone #

Rotation. A swaying vehicle also yaws, and a camera fixed to it turns. A turn moves the epipole across the picture without adding any baseline, and it was not modelled here; the sway here is a pure sideways displacement.

Where the pictures are. The pictures’ positions are known exactly. A real sequence recovers its own camera positions from the same pictures, and a point is a line over there found how much a pose that is slightly wrong costs a single pair; with a sway of a centimetre, a pose error of a few millimetres is the same size as the baseline the centre depends on.

A scene at more than one distance. Every cost is for a surface at a stated distance, 4, 8 or 16 m. A real view ahead holds the road a few metres off and the horizon at infinity in one picture, and the centre of the frame is usually the far end of that range; the cost there grows with the depth, so the blind centre is blindest exactly where a forward-looking camera is pointed.

Pixels read to better than a pixel. Every number is per pixel of reading error and scales in proportion to it. A matcher good to a tenth of a pixel divides every cost by ten, and brings the sway needed for a tenth of the depth at 8 m down to 1.4 cm.

Still open: what a turn of the vehicle does to the disc #

The sway measured here is a pure sideways step. A vehicle that changes lane also turns, briefly, and a camera fixed to it turns with it, which moves the epipole across the picture — for a turn of a degree, by the focal length times the tangent of a degree, about ten pixels on this camera — without adding any baseline at all.

The measurement that settles what that does takes the same three pictures with the middle one turned by a stated yaw instead of, and then as well as, swayed, and asks two things: whether a turn alone changes the depth of the centre mark at all, since it moves where the epipole sits without making the rays cross, and whether a turn and a sway together can put the blind point somewhere harmless — off the frame, or onto the sky — which a steered camera could then arrange on purpose.