Visual Plane
every picture states the point it is correct from
Surfaces that are not flat

Undoing a picture made on a curve

Three rounds of work here have measured what a curved receiving surface costs going forward — 5.67 mm on a dish, 9.07 mm on a ridge, 529.4 mm on a vault. None of them asked whether the design can be got back. It can, exactly, and the price is stated precisely: you have to supply the surface. Told the floor, the recovery returns a design to 1.1e-12 mm; told nothing, and fitting the four marks every rectification tool fits, it is 111 mm out.
19 min read 12 figures The surface is a choiceThe round tripFitted, not assumed

Worth reading first: When the picture surface is not flat · The floor that is not a plane.

Three separate arguments on this site have measured what happens when a picture is made on a surface that is not flat. A shadow cast onto a dished floor departs from any homography by 5.67 mm, and onto a ridged one by 9.07 mm. An anamorph laid out for a vault instead of a flat floor differs from the flat design by 529.4 mm. Every one of those is a measurement of the forward direction: put something onto the surface and see how far it is from what a plane would have given.

None of them is the question a reader with a photograph actually has, which is the other way round. There is a design on that curved floor. Can it be got back?

A rectangular grid, painted on a ridged floor and photographedEvery mark is where a flexible rule laid along the floor would put it, so the grid is exactly rectangular on the surface and exactly 1800 × 1200 mm when the floor is unrolled. Undone with the floor known, the design comes back to 1.1e-12 mm. Undone by a homography fitted to the four ringed marks — which is what a rectification tool does — it is exact at those four and 111 mm out at the worst of the others.111 mm outcorrect from 21 cm, at 160 mm wide1e-12 mm with the floor · 111 mm without
Fig. 1 A rectangular grid painted on a ridged floor, photographed. Every mark is where a flexible rule laid along the floor would put it, so the grid is exactly rectangular on the surface. Undone with the floor known, it comes back to 1.1e-12 mm; undone by a homography fitted to the four ringed marks, it is exact at those four and 111 mm out at the worst of the rest.

The answer is yes, exactly, and the price is a single sentence: the surface has to be supplied. What follows is what that means, what it costs not to, and what it means for the recovery to be exact at all.

What “the design” means on a curved floor

Before anything can be recovered, it has to be said what is there to recover, and on a curved surface that is not obvious.

Take a signwriter laying out lettering on a floor that dips. They do not work from a plan; they work with a flexible rule laid along the surface, measuring distances on the floor. The result is a design that is exactly rectangular when measured that way, and not rectangular at all when measured on the plan.

That is the object this essay recovers: the design in the surface’s own coordinates, which are arc lengths along it. Those coordinates exist for the ridge — a ridged floor can be unrolled onto paper without stretching, so there is a genuine flat design and it is unique — and they do not exist for every surface. Which floors have them is a question with a sharp answer and an essay of its own; for now the ridge does, and that is what makes the recovery well posed rather than a matter of choice.

A ridged floor, and the same floor laid out flatThe section through the ridge is a parabola 2.6 m wide across the plan and 2.6105 m long along the surface. Unrolling it is exact: the marks below are the marks above, each moved to its own arc length, and the map between them stretches nothing — the worst strain across the patch is 7.4e-9. Reading the floor off its plan instead stretches it by 0.86%.the floor, in placeunrolled — 2.611 m of surfacea ridged floor, k = 0.062.6 m of plan is 2.611 m of floor
Fig. 2 The unrolling the design is measured in. The floor’s section is a parabola 2.6 m across the plan and 2.6105 m along the surface, and the marks below are the marks above moved to their own arc lengths. The map between them stretches nothing.

The recovery, in three exact steps

Given a photograph and the floor, the undo is a composition of three maps, none of which is a fit.

From the picture to a ray. An image point names a direction from the eye: the camera’s basis, its focal length, its principal point. Exact.

From the ray to the surface. Follow the ray until it crosses the floor. For a shape given as a height field this is a root-find rather than a formula — deliberately, because the family of floors here includes a step, which is not a quadric anybody wants to solve by hand, and a solver that only handled quadrics would have quietly excluded it.

From the surface to the design. The point on the floor has a position in the floor’s own arc-length coordinates. Exact, in closed form for the ridge: the arc length along a parabola integrates to

u(x)=12x1+4k2x2+14kasinh⁡(2kx),u(x) = \tfrac{1}{2}x\sqrt{1+4k^2x^2} + \tfrac{1}{4k}\operatorname{asinh}(2kx),

written out rather than integrated numerically, because it is the thing being claimed exact and a quadrature’s own error would be indistinguishable from the surface failing to unroll.

Compose the three and the design comes back at 1.1e-12 mm — which is arithmetic and not accuracy. There is no least-squares step anywhere in the chain, so there is no residual to report and no conditioning to worry about. That is the sense in which knowing the surface is enough: it is not that the recovery becomes good, it is that it stops being an estimate.

The ground, turned into a planFour corners of a rectangle of known proportions fix the homography. Three lengths it was never given come back to 1e-15 relative — so every measurement on that plane is available, in units of the rectangle's own width.the picturethe same plane, rectifiedacross — 0.6000 widths (true 0.6000)along — 0.7667 widths (true 0.7667)diagonal — 1.2023 widths (true 1.2023)worst error 1e-15 relativethe probes were not used to build the map
Fig. 3 The flat case, for comparison, and the reason the curved one feels like it should be harder. On a plane the whole recovery is one 3×3 matrix from four corners, and three lengths it was never given come back to arithmetic noise. What changes on a curve is not the accuracy but the amount that has to be supplied.

And what it costs to guess

The interesting number is the other one: what happens to somebody who has the photograph and no plan of the building.

They do what every rectification tool offers. Mark four points whose positions in the design are known — four corners of a panel, four registration marks — and fit the homography that carries them there. Four correspondences determine one exactly, so this always works, and it is exact at the four.

Everywhere else it is 111 mm out, on a design 1800 mm across. Six per cent of the width, from a floor whose deviation from flat is a few centimetres.

One photograph of one floor, undone three waysThe design is 1800 mm across. Knowing the surface returns it exactly — nothing is fitted, so there is no residual to report beyond arithmetic. Assuming the floor is flat is exact at the four marks the homography was given and 111 mm out elsewhere. And knowing the shape but getting its curvature 10% wrong costs 11.0 mm, which is the price of the parameter rather than of the shape.what the recovery was toldworst error in the recovered designthe surface, known1.1e-12 mmassumed flat, four marks110.97 mm6e-13 mm at the fourthe surface, curvature 10% out11.00 mma a ridged floor, k = 0.06, design 1800 mm wide1e-12 mm · 111 mm · 11.0 mm
Fig. 4 One photograph, three answers. The exact recovery, the flat fit, and a middle case where the shape is known and its curvature is not. The flat fit’s error at its own four marks is the control: a version of this reporting a large residual there would be reporting a broken fit rather than a curved floor.

That the flat fit is exact at its four marks is the control, and it matters. Without it, the 111 mm could be a badly conditioned solve rather than the shape of the floor. With it, the four marks come back to a hundredth of a micron and the rest do not, which localises the error precisely: the map is right where it was told and wrong where it was asked.

Two practical notes come out of the same figure, and both were errors before they were notes.

Choose the four marks at the corners. The first version of this recovery picked them at evenly spaced indices through a grid stored row by row, which selects four points down a diagonal — very nearly collinear — and the homography fitted to them reported the flat rectification as 4.18 m out. That number is about the sampling, not the floor. Four nearly-collinear correspondences is the same trap a shadow’s four-point fit met here before, and it does not announce itself: the solve succeeds and the answer is wild.

And a curved floor is not a wide-angle lens. The error above is not a distortion of the picture. It is a mismatch between two surfaces, and no amount of undistorting the photograph touches it.

What the receiving surface costs a four-point fitThe same lamp and the same occluder, cast onto four surfaces. Four marks of the shadow are matched to four known points of the occluder, the map they determine exactly is built, and the other sixty-eight points are predicted by it. On the plane the prediction is right to 3e-13 mm; on the others it is not, and the fit is still exact at the four everywhere — 2e-13 mm — which is what makes the rest a prediction rather than a residual.a flat floor3e-13 mma homology — four points determine ita dished floor5.67 mmnot a homologya ridged floor9.07 mmnot a homologya floor with a step74.95 mmnot a homologyworst error of the predicted pointsexact at the four fitted, on every surface
Fig. 5 The forward measurement this essay inverts, on the same four floors. One lamp, one occluder, four surfaces: a plane at arithmetic noise, a dish at 5.67 mm, a ridge at 9.07 mm, and a step — which is two planes, each of them exactly a homology — worst of all.

Why the flat fit is wrong in the way it is

It is worth knowing the shape of the error, because it explains why it cannot be tuned away.

A homography of the plane is a map with eight parameters. The map from the photograph to the design, on a curved floor, is not in that eight-parameter family at all — it is a ray trace through a surface, and the surface has as many parameters as its shape does. Fitting a homography to it is not fitting an imperfect model; it is fitting a model of the wrong kind.

The tell is the pattern of residuals. Model error of the ordinary sort is noisy and shrinks when more data is used. This one is smooth, systematic, and grows toward the parts of the floor furthest from the four fitted marks — because the fit interpolates between them and extrapolates beyond them, and the surface’s departure from flat is a smooth function that the eight parameters have no way to absorb.

Adding more marks does not help either, and that is the diagnostic. A least-squares homography over twenty marks spreads the error more evenly and does not reduce its size, because there is no homography that fits. The residual after fitting twenty is a measurement of the floor.

One map, applied across the creaseA homography fitted to four marks of the shadow that all lie on the floor, then run backwards over the whole shadow to recover the card. Where the shadow is on the floor it returns the card to 6e-16 m. Where the shadow has climbed the wall it returns an outline up to 78.9 cm away from the card — not because the map is inaccurate but because it is the wrong map, and nothing in the picture of the shadow says where one stops and the other starts.recovered from the shadow — up to 78.9 cm out on the wallcorrect from 16 cm, at 160 mm widefloor 6e-16 m · wall 78.9 cm
Fig. 6 The same failure at its sharpest, on a floor-to-wall crease. A map fitted to four marks on the floor and applied across the whole shadow returns the occluder to arithmetic noise where it was fitted and up to 78.9 cm out on the wall. Piecewise projective is not projective.

A picture surface and a receiving surface

There is a symmetry here worth naming, because it turns this essay into a special case of something the site has been circling for a long time.

The curved field is about the surface a picture is formed on — a cylinder, a sphere, a fisheye’s focal surface. This essay is about the surface a picture is painted on before a camera looks at it. Those sound like different subjects and they are the same one seen from two ends.

In both cases the question is: given a map from directions to a two-dimensional thing, what does the two-dimensional thing being curved do to what can be recovered? And in both cases the answer is that the surface has to be supplied, because it is not in the picture.

A photograph of a fisheye picture does not carry the fisheye’s rule; a photograph of a painted floor does not carry the floor. Both are recoverable only from something else — a specification, a calibration, a plan — and both are exactly recoverable once supplied.

One room at 120° across, cast onto six picture surfacesEach panel holds the same angular width of the same scene. The flat plane keeps every straight line straight and pays for it at the edges; the five curved surfaces bend the ground lines by 3.5% to 6.0% of their own length.plane — bend 0.0%cylinder — bend 6.0%stereographic — bend 3.5%equidistant — bend 4.5%equal-area — bend 5.1%equirect. — bend 5.7%120° across in every panelsame scene, same angle, six surfaces
Fig. 7 The other end of the symmetry. Six picture surfaces given one scene and one field of view, each doing something different to it — and each of them exactly invertible, provided you are told which one you are looking at.
The vault refuses the projective descriptionThe same design and the same eye, cast onto a flat floor and onto a barrel vault of radius 4.0 m. The best homography fitted to the floor's marks misses by 1.4e-15 m; fitted to the vault's it misses by 529.4 mm, which is 7.7% of the marks' own extent, and no choice of four marks helps.barrel vaulteye · 1.62 mon the floor1e-12 mmon the vault529.4 mmworst miss of the best homography, log scalevault radius 4.0 m7.7% of the extent, against 4e-14%
Fig. 8 And the case where the two ends meet. A design laid out on a vault so that it reads correctly from one eye: the receiving surface is curved, the picture surface is the retina, and the 529.4 mm between the vault’s design and the flat one is the whole subject of this essay measured forward.

Why the ray trace has to be a root-find

A small point of construction that turns out to carry a claim.

The second step — from a ray to the surface — is done by bisecting on the signed height difference along the ray, rather than by solving a quadratic. For a dish or a ridge a closed form exists and would be quicker. It is not used, and the reason is the step.

step is two horizontal planes with a jump between them. It is developable, its plan is its development, and it is nonetheless the surface that broke the four-point fit worst of the four when a shadow was cast onto it. A solver written for quadrics would have had to special-case it or leave it out, and leaving it out is how a family of surfaces quietly becomes a family of smooth surfaces without anybody deciding that.

The bisection also produces the refusals honestly. A ray that never crosses the floor returns nothing rather than an extrapolated crossing, which is what a map on this site is required to do at its own boundary. One of those refusals turned out to contain a fact worth knowing: a ray aimed above the horizon meets no flat floor, and it does meet a ridged one, a long way off — a ridged floor is a parabolic trough, so it rises without bound at the sides. “Above the horizon” is a guard that belongs to the plane and to no other surface.

What has to be known, and what does not

The recovery needs the floor. It is worth being precise about which parts of “the floor”.

The shape, as a function. Not a rough idea of it: the actual height field, because the ray trace uses it pointwise.

The camera. Focal length, principal point, position and orientation, since the first of the three steps is the picture-to-ray map. In practice this is recovered from the same photograph by ordinary means, and the essays on the camera coming back out of its own picture are the route.

And nothing about the design. No known lengths, no assumed rectangle, no registration marks. That is the part worth noticing: the flat fit needs four known correspondences and gets the answer wrong; the exact recovery needs none and gets it right. The information was never in the marks, it was in the floor.

A box drawn from a known camera, and the camera recovered from the drawingThree vanishing points found from the twelve drawn edges alone give back the focal length to 2e-15 relative.recovered principal pointused to drawrecoveredgapfocal length898.76898.762e-15principal x345.0345.06e-13angle42.0°42.0°—correct from 21 cm, at 160 mm wide42° across
Fig. 9 Where the camera half comes from, when it is not written down. Three vanishing points found in the picture’s own edges give back the focal length and the principal point, with nothing about the scene supplied.
One photograph of one floor, undone three waysThe design is 1800 mm across. Knowing the surface returns it exactly — nothing is fitted, so there is no residual to report beyond arithmetic. Assuming the floor is flat is exact at the four marks the homography was given and 111 mm out elsewhere. And knowing the shape but getting its curvature 30% wrong costs 32.8 mm, which is the price of the parameter rather than of the shape.what the recovery was toldworst error in the recovered designthe surface, known1.1e-12 mmassumed flat, four marks110.97 mm6e-13 mm at the fourthe surface, curvature 30% out32.75 mma a ridged floor, k = 0.06, design 1800 mm wide1e-12 mm · 111 mm · 32.8 mm
Fig. 10 The middle row again with the assumed curvature further from the truth. The shape is right and the parameter is not, and the error scales with the mistake — which is the next question, and it has a number.

What a reader with a photograph should do

The result is worth turning into a procedure, because the wrong procedure is the default one everywhere.

Ask whether the surface is flat, and answer it with a measurement rather than a glance. A floor that dips by a few centimetres over two metres looks flat in a photograph and costs six per cent of the design. The relevant question is not whether it looks flat but whether its departure from flat, divided by the distance to the camera, is small compared with the accuracy wanted — and for a floor photographed from three metres, a two-centimetre dip is not.

If it is flat, four marks are the whole job, and the rectification is exact.

If it is not, get the surface — from a plan, a survey, a laser scan, or by fitting it from the same photograph if there is enough in the scene to fit it from. Then the recovery is exact and needs no known correspondences at all.

And check the residual at the marks the fit was not given. That is the one diagnostic that separates the two cases without any prior knowledge. Fit a homography to four marks, predict a fifth, and see how far off it is. On a flat floor it lands; on a curved one it does not, and the size of the miss is the floor.

That last test is the same one the shadow work uses on its receivers, and it is the cheapest thing in this essay: one extra mark, and the answer to whether any of the rest is needed.

The three levels, and the one that is a refusal

Written out, the answer has three levels of knowledge and they differ in kind rather than in degree.

Know the surface and the recovery is exact. No fitting, no residual, arithmetic only.

Know the shape but not its parameter and the error is proportional to the mistake, with a constant that can be measured and turned into a tolerance.

Know nothing and assume flat, and the error is the shape of the floor, exact at whatever was fitted and wrong everywhere else.

And there is a fourth level which is not a worse version of the third but a refusal. For a floor that cannot be unrolled — a dish, a dome, anything with curvature — there is no flat design to recover, because “the flat design that was painted on it” names nothing. A recovery computed through some chosen flattening of a dish returns the choice, not the design.

Which floors those are is decided by a single number, and it has nothing to do with how curved the floor looks.

Gaussian curvature decides whether a floor can be unrolled at allA surface can be laid flat without stretching exactly when its Gaussian curvature is zero everywhere — Gauss's theorem, which says that quantity survives any bending. Three of these four floors have none, including the ridge, which curves visibly. The dish has 0.0144 per square metre, and no cleverness in the flattening removes it.floorGaussian curvature, worst over the patchcan it be unrolled?a flat floor0yes, exactlya ridged floor0yes, exactlya floor with a step0yes, exactlya dished floor0.0144by nothing whateverfour floors, k = 0.06three at zero, one at 0.0144 m⁻²
Fig. 11 The number that decides it. Three of these four floors can be laid flat without stretching, including the ridge, which curves visibly; the dish cannot, and no cleverness in the flattening removes that.
How well the floor has to be known, in per centThe recovery is told a curvature that is wrong by a stated fraction, and the worst recovered mark is measured. The line through the origin is very nearly straight — the error per unit of misknowledge varies by 1.4% across a twentyfold range — so a tolerance can be quoted: 0.91% of the curvature buys 1 mm on a design 1800 mm wide.025507510010203040how wrong the assumed curvature is, as a % of the true oneworst error in the recovered design (mm)0.91% buys 1 mma a ridged floor, k = 0.0644.0 mm at 41% out
Fig. 12 And the tolerance the middle level buys. The error rises very nearly linearly with the mistake in the assumed curvature, so a specification can be quoted rather than guessed: about nine parts in a thousand of the curvature buys a millimetre on a design 1800 mm wide.

Where this goes next

a flat floor3e-13 mma homology — four points determine ita dished floor5.67 mmnot a homologya ridged floor9.07 mmnot a homologya floor with a step74.95 mmnot a homologyworst error of the predicted pointsexact at the four fitted, on every surface The floor that is not a plane A shadow on a flat floor is a homology, so four marks determine the whole map and the rest of the outline comes back exactly. Dish the floor and the same four marks mispredict the rest by 5.67 mm; ridge it and 9.07 mm; put a step in it — two planes, each of them exactly a homology — and 74.95 mm. The receiver's shape is what breaks the projective description, and it breaks it worst where the surface is flattest. the picturethe same plane, rectifieddoor — 0.5068 widths (true 0.5068)sill — 0.3026 widths (true 0.3026)diagonal — 1.1766 widths (true 1.1766)worst error 4e-16 relativethe probes were not used to build the map Flattening a façade out of the photograph Four corners of a rectangle whose proportions are known are enough to undo the projection of one plane. After that the plane can be measured with a ruler — lengths, angles, areas, all of it — in units of the rectangle's own width, and lengths the map was never given come back to fifteen digits. d = 0d = 0.5d = 1d = 2d = 4d = 0straight48.06° angle×17.80 aread = 0.54.81% bend23.08° angle×4.66 aread = 17.21% bend16.84° angle×2.65 aread = 29.61% bend23.13° angle×1.57 aread = 411.54% bend32.71° angle×1.73 areaa general straight line · worst angle · area range, over the field130° acrossangular minimum at d = 1.00 One parameter between two surfaces Wide architectural views are usually made on a projection with a number attached to it — a family running from the flat plane at one end toward the cylinder at the other, with everybody using the value one. That value has never been given a geometric defence. Measured across the family on this site's own battery, the worst angular error over the field has a minimum, and the minimum is at 1.04.

Reads more easily once this is understood

Essays that name this one as worth reading first.

Shares its objects with

Essays that name at least two of the same things, and that neither author linked.

Named objects

A flat tag is an object no other essay names yet.

AnamorphosisArc lengthDevelopableGaussian curvatureHomographyInvertibilityIsometryleast squaresPicture planeProjective mapRay tracingre-projectionRectificationsingle-view metrology