Every row is a different camera

Almost every digital camera in existence reads its sensor a row at a time. The top row is read first, the bottom row last, and between the two the world has had a few milliseconds to move.

The usual account of this stops at the artefact: a photograph of a moving car has leaning wheels, a propeller becomes a set of curved blades, a picture taken from a vibrating helicopter is a wobbling ribbon. All true, all well known, and all a description of symptoms.

The statement worth making is structural, and this site has already made it about a different object. A rolling shutter is a pushbroom in time. The cultures phase modelled a Chinese handscroll as an eye that translates while imaging one line at a time; a rolling shutter is an eye that translates while imaging one line at a time. The roll runs down the picture instead of along it and everything else is identical — including, exactly, the battery of measurements that phase built.

The model, and the equation it makes implicit #

Row $r$ of $R$ is exposed at time $t = (r/R)\,T$ for a readout time $T$. So the camera that imaged a world point is the camera at the instant that point’s own row was read — and finding where the point lands is therefore an implicit problem, because the row it lands on decides which camera drew it.

$$q = \Pi\big(p;\ C(t)\big), \qquad t = \frac{q_y}{R}\,T$$

Two equations, and the unknown appears on both sides. The standard solution is a fixed-point iteration: guess a time, project, read the row, take the time that row implies, repeat.

It converges quickly and it does not always converge, and the failure is worth naming because it is a real regime rather than a numerical nuisance. When the image moves down the frame faster than the readout sweeps down it, a point can be caught by two rows or by none, and the iteration has no fixed point to find. The solver returns nothing in that case rather than a wandering answer — the same refusal this site’s camera makes for a point behind the eye, and for the same reason.

The lean, and the three numbers it comes from #

A vertical line under a lateral pan is drawn leaning, and the lean is

$$\tan\varphi = \frac{f\,v\,T / z}{R}$$

— image speed in pixels per second, times the readout, over the height of the frame.

Nothing about the line is in that expression. Its height, its position across the frame and its length are all absent, so every vertical in the picture leans by the same angle, and the artefact is a shear of the whole frame rather than a distortion of one object. That is the diagnostic that distinguishes rolling shutter from motion blur, from lens distortion, and from a genuinely leaning object.

The gate does not assert the lean against a threshold. It asserts a doubling law: twice the speed is exactly twice the tangent, to nine digits. That is a stronger claim than agreement at one setting, and it says the shear is the travel rather than some property of the scene that happens to scale.

The first version of that check did use a threshold — “the lean is more than a fifth of a degree” — and it failed at the slow end of the figure’s own slider while the claim it was defending stayed true. A bound on the size of an effect is a bad way to assert a law, because the law holds where the effect is small and the bound does not.

The measurement the cultures phase already built #

The cultures phase’s central question about any drawing system was: has it got a centre? Fit the best single point to the system’s own rays and report the miss. A pinhole’s rays meet at $4 \times 10^{-16}$ m; a handscroll’s miss by 7.97 m over 26.5 m of track; and the phase’s finding was that the miss is the standard deviation of the eye’s track.

Run the identical solver — the one written for a refracted picture, reused for a handscroll, and now on its third object — over the rays of a rolling frame. Twenty-eight world points, each imaged by whichever row happens to see it, each ray leaving from wherever the eye was at that row’s instant:

• the rays miss by 16.07 mm;
• the eye’s track over the readout has a spread of 16.26 mm;
• miss ÷ spread = 0.988;
• and with the motion removed, the same points through the same solver miss by $2.1 \times 10^{-16}$ m.

So a rolling frame is not a projection of anything from anywhere, by the same measurement and to the same law as a scroll. It is very nearly a projection, because 16 mm is small compared with a scene — which is precisely why the artefact is usually described as a distortion rather than as a failure of the picture to have a viewpoint at all.

Why the agreement is 0.988 and not fifteen digits #

The scroll’s version of this law is exact to fifteen digits — 7.9738 m measured against 7.9738 m predicted. This one is right to about one part in a hundred, and the difference is a real property of the two objects rather than a slack tolerance.

A scroll’s rays leave the track at right angles by construction: each column images the single vertical plane the eye is level with, so the ray and the track are perpendicular and the whole of the track’s spread contributes to the miss.

A rolling shutter’s rays leave at whatever angle the scene puts them at. Their perpendicular components carry only part of the track, so the miss is a little less than the spread — 0.988 of it, for this arrangement. Change the arrangement and that number changes a little; it is bounded above by 1 and is never zero unless every ray happens to be parallel to the track, which is a scene with nothing in it.

That is worth writing down as a difference rather than smoothing over, because the site’s habit is that a law with a stated regime is worth more than a law quoted at one setting. The scroll’s version is the special case; this is the general one.

The readout is a shutter and a clock at once #

One implementation detail is worth lifting out because it decides the size of everything above.

The readout time and the exposure time are independent. A sensor can expose every row for 1 ms and still take 30 ms to read them all, because reading is a separate operation from collecting. So the shear is governed by the readout and the blur by the exposure, and a photographer changing the shutter speed changes one and not the other.

That is the practical reason the artefact resists the obvious fix. Everything a person can adjust on a camera — aperture, shutter, sensitivity — leaves the readout alone, because the readout is a property of the sensor’s electronics. The only settings that touch it are the ones that change how much of the sensor is being read: a smaller crop, fewer rows, or a lower resolution, all of which are faster and all of which are usually chosen for other reasons.

Which is why the artefact is described in the trade as a property of a camera rather than of a shot. It very nearly is.

The four cameras, with a fifth cell now filled #

A scroll is not a panorama sets out a two-by-two: an eye can hold or move its position, and hold or turn its direction. Fixed and fixed is an ordinary camera; fixed and turning is a panorama; moving and fixed is a scroll and a satellite; moving and turning is not a picture at all but a reconstruction problem.

A rolling shutter belongs in the third cell, and it adds something the other occupants do not have: the eye’s motion is not part of the system’s definition. A scroll’s eye travels because a scroll is a picture made by a travelling eye; that is what the object is. A rolling shutter’s eye travels because somebody happened to be walking, and if nobody is walking the object is an ordinary camera.

That is the distinctive thing about this member of the family. It is a drawing system conditionally — it becomes a pushbroom only when something moves — and the same hardware produces an exact projection and a centreless one depending on circumstances entirely outside itself. A scroll cannot be persuaded to have a centre; a rolling shutter has one whenever the world is still.

Rotation and translation give different artefacts, for the reason the scroll field found #

The cultures phase’s sharpest finding about the scroll was that curvature and centrelessness are independent. A rotating eye bends straight lines and keeps its centre exactly; a translating eye bends them and has none.

A rolling shutter shows both halves of that, from one piece of hardware:

Panning — the camera turns on the spot. The eye does not move, so the frame still has a centre, and the artefact is a pure shear: verticals lean, everything keeps its shape, and the picture is a projection of the scene from one point with a sheared image plane. Stitching such frames works.

Translating — the camera moves. Now there is no centre, the artefact is depth-dependent, and near and far objects shear by different amounts because $f v T/z$ has a $z$ in it. That is the same depth-dependence that makes a panorama shot about the wrong point unstitchable, and it has the same cure, which is not to do it.

So the two artefacts look alike and are not alike, and the test that separates them is the one the scroll field earned: fit a centre and see whether it exists. That test does not care what the picture looks like.

What the artefact is not #

Three things it gets confused with, each of which has a different signature, and separating them is most of the practical value of having a model.

It is not motion blur. Blur integrates over an interval; a rolling shutter samples at different instants. A frame can have one, the other, or both — a short exposure with a long readout gives sharp sheared geometry, which is the case that looks most like a mistake in the drawing. And the two have opposite dependences on shutter speed: shortening the exposure removes blur and does nothing whatever to the shear.

It is not lens distortion. A lens bends lines by an amount depending on their distance from the centre of the field, so the effect is radial and symmetric and identical in every frame. A rolling shutter shears every vertical by the same angle regardless of position, and the angle changes from frame to frame with whatever is moving. Radial-and-constant against uniform-and-varying: the two are as different as two artefacts can be, and they get the same word because both are described as “the picture is bent”.

It is not perspective. A leaning building in a photograph is usually a real vertical vanishing point doing exactly what it should. That leaning converges — verticals meet at a point — while a rolling shutter’s leaning is a shear, and parallel verticals stay parallel under a shear. Whether the leaning lines meet is the test, and it needs one glance at two of them.

That third one is worth having explicitly because it is the confusion that produces wrong corrections. Straightening a genuinely converging vertical destroys a true projection; straightening a sheared one restores it. The two look alike in a single frame and are told apart by asking whether the lines are parallel.

What can be undone and what cannot #

The practical question is whether a rolling-shutter frame can be corrected, and the answer follows from everything above rather than from experiment.

A pure rotation can be undone exactly, given the rotation. Each row is a projection from the same centre through a different orientation, so mapping row $r$ back through the inverse of its own rotation gives a frame from one centre — and that is a homography per row, with no depth in it. It is exact because a rotating eye keeps its centre, which is the whole content of the scroll field’s separation.

A translation cannot be undone without depth. Each row is a projection from a different point, and mapping between two different centres requires knowing how far away everything is. Which is what a second view supplies and one view does not, so the correction needs either a depth estimate or a scene assumption, and the standard assumption — that the scene is flat and far away — is exactly the assumption that makes a translation look like a rotation.

That is a clean and slightly uncomfortable result: the correction that is applied in practice works by pretending the case that cannot be corrected is the case that can. It works well when the assumption nearly holds and fails in exactly the situations where the artefact is worst, which is close subjects and fast motion.