Visual Plane
every picture states the point it is correct from
Through water and glass

A picture through water has no viewpoint

Continue the rays of a refracted picture into the water and fit them to a common point. They miss it by ten millimetres. The same fit with the water taken away misses by zero, which is what makes ten millimetres a measurement rather than a number.
16 min read 7 figures Not a projection at allThe round tripWhat a projection destroys

There is a way of asking whether something is a projection that does not depend on knowing anything about the thing being projected.

A projection through a centre is exactly the statement that every ray of the picture passes through one point. Not approximately, not on average — one point, and the same one for every ray. So take the rays of a picture, continue them into the space where the objects are, and ask whether they meet. If they do, the picture is a projection and the meeting point is the centre. If they do not, it is not, and how badly they miss is the measurement.

Applied to a picture taken through water, the answer is that they miss by ten millimetres.

The rays of a refracted picture, continued into the waterEvery ray leaves the pinhole, bends at the surface and carries on. Fitted to a common point they miss it by 9.9 mm — the circle is that miss drawn at the figure's own scale. With the water removed the same fit misses by 0e+0 m.the water surfacethe pinholethe rays miss by 9.9 mmno single viewpoint — the rays miss by 9.9 mmdry control: 0e+0 m
Fig. 1 Eleven rays leaving one pinhole, bending at the surface, and continuing into the water. The circle is the RMS miss of the least-squares common point, drawn at the figure’s own scale rather than quoted in a caption — so the failure is a size on the page.

The fit, and the control beside it

The closest point to a set of lines is a small least-squares problem. For rays with unit directions di\mathbf{d}_i passing through points pi\mathbf{p}_i, the point minimising the sum of squared perpendicular distances solves

∑i(I−didiT)x=∑i(I−didiT)pi\sum_i \left(I - \mathbf{d}_i\mathbf{d}_i^{\mathsf{T}}\right)\mathbf{x} = \sum_i \left(I - \mathbf{d}_i\mathbf{d}_i^{\mathsf{T}}\right)\mathbf{p}_i

which is a 3×3 symmetric system, solved here by elimination with partial pivoting. It refuses rather than returning a number when the bundle is degenerate — every ray parallel, so the closest point is at infinity — and that refusal is fired at in the site’s gate rather than trusted.

What the figure reports is the RMS miss and the worst miss, in metres of the world.

The number that makes those mean anything is the one computed beside them: the same fit, on the same angles, with the water taken away. That comes out at exactly zero, because the rays really do all pass through the eye. A measurement of 9.9 mm quoted without that control would be a number; quoted with it, it is a statement about the geometry.

This is the shape of every claim in this field, and it is worth naming once. The refracted case is always drawn against the pinhole case, computed by the same code, from the same scene. Anything else would be reporting the machinery.

What the rays actually do

The miss is not random scatter. The rays converge to a surface rather than to a point, and the surface has a name in optics — the caustic — and a shape that is easy to describe once the arithmetic is in view.

A ray leaving the pinhole at a small angle bends very little at the water surface and its backward extension, in the water, points at a virtual position close to h/nh/n below the surface. A ray leaving at a large angle bends a great deal, and its backward extension points at a virtual position much shallower. So the virtual positions the rays appear to come from are strung out along a curve, running from h/nh/n on the axis toward the surface at the rim. That curve is the same object as the apparent depth falling with the angle of view, seen from the other side: the pool’s apparent bottom is not a plane because the eye’s rays have no common origin in the water.

The least-squares fit hands back the point that is least wrong for all of them at once, which is somewhere in the middle of that curve, and the RMS miss is the spread of the curve about it. Pushing the surface further from the eye spreads the rays over a wider range of angles before they reach it and the miss grows; pulling it closer shrinks it. The slider is that.

The invariant

The miss in metres says the picture is not a projection. The next question is which of the theorems built on projections still hold, and there is a natural first one to ask, because the whole site rests on it.

Four collinear points in the world have a cross-ratio, and a projection preserves it exactly. It is the one quantity that survives, and almost everything checkable about a picture on this site is checked with it: the depth of a row of posts, the height of a building from a photograph, whether a taught construction depicts what it claims to.

The cross-ratio of four points, dry and wetThe pinhole returns the world's cross-ratio to 2e-16. The same four points seen through 0.40 m of air and then water return it 1.20% out, and stop being collinear by 1.24 px.pinholethrough waterin the world1.353437through the pinhole1.353438through the water1.369733no single viewpoint — the rays miss by 1.24 px of bendcross-ratio 1.20% out, against 2e-16 dry
Fig. 2 The same four collinear world points, imaged through a pinhole and through 0.40 m of air and then water. The pinhole returns the world’s value to 2e-16. The water returns it 1.20% out, and the four points stop being collinear by 1.24 px.

One and a quarter per cent does not sound like much until it is put where such a number is used. A height recovered from a photograph is a cross-ratio; so is a depth along a receding row; so is the check that says a drawn box is a box. A per cent and a quarter on the invariant is a per cent and a quarter on the answer, at best, and worse where the ratio is near a singularity.

There is a second failure in the same figure, and it is the cleaner one. The four points stop being collinear. A projection sends lines to lines — that is not an accident of the definition, it is what makes vanishing points exist at all — so a set of four world points on one line whose images are 1.24 px off a straight line has been through something that is not a projection, and no further argument is needed.

One pixel dry, two pixels wetA near point and a far one on the same ray of the pinhole camera are the same image point to 6e-14 px. Through the tank they are 149.1 px apart, because the displacement a layer adds is a length and a length matters more to a near point than to a far one.where the two points landthrough the tank: 149.1 px apartthrough nothing: one marknear point, 0.9 m135.6 pxfar point, 6.0 m284.7 pxthe two, apart149.1 pxat 34° off axis, through 12 mm of glass into waterno single viewpoint — the rays miss by 149.1 px of splitno warp of the image can undo a depth-dependent shift
Fig. 3 The sharper version of the same failure. Two world points on one ray of the pinhole camera are one image point — that is the definition of a projection, and the figure confirms it to 6e-14 px. Through the tank they are 149.1 px apart.

Two failures of different kinds

It is worth separating what the two measurements say, because they are not the same complaint.

The cross-ratio being wrong says the map is not projective. That is a statement about the map as a function on the image: no homography, no change of camera, no rectification will repair it, because those are all projective and a projective map composed with a projective map is projective. The image would have to be warped by something else.

The rays having no common point says something stronger. It says the map is not a function of image position at all. The correct place to send a point depends on how far away that point is, so even a general non-projective warp of the image cannot repair it: the information needed is not in the picture. The near-and-far demonstration in the previous essay is the sharp form of this — two world points that share one image point in air land 102.2 px apart in water.

That distinction is why underwater photogrammetry is genuinely hard rather than merely fiddly. The standard fix is not to correct the image but to model the interface: put the water surface, the port, and the indices into the ray-tracing itself, and stop pretending there is a camera matrix. Everything above is the argument for why there is no shortcut.

Four points on a line, before and after a projectionLength and the ratio of lengths do not survive the projection; the cross-ratio does, agreeing to 0e+0 relative.horizonABCDon the groundin the picturelength AB1.00011.3930ratio AB:CD0.56670.6837cross-ratio1.31681.3168correct from 26 cm, at 160 mm wide34° across
Fig. 4 The control this field is measured against, from the site’s first field: the same invariant through a plain projection, surviving to fifteen digits. Everything in this essay is a departure from that figure.

What can still be measured, and how

None of this makes underwater measurement impossible; it makes it a different problem, and the shape of the different problem follows from the same arithmetic.

Ratios along one plane parallel to the interface still behave. Points at a single depth are all displaced by the same rule as a function of angle, so a homography fitted to that plane alone is a good approximation over a limited field. That is why a diver’s ruler laid flat on the bottom, photographed from directly above, gives usable measurements — the configuration removes both of the problems at once, the angles being small and the depth being constant.

Anything spanning depth does not. As soon as the scene has extent along the optical axis, the depth-dependence bites, and the only honest route is the ray-based one: trace forward through the modelled interface, and solve for the scene that reproduces the observed image.

And the interface has to be modelled, not fitted away. It is tempting to absorb the whole effect into a radial distortion coefficient, because the pattern looks radial and there is a well-known polynomial for that. It does look radial, and over one plane at one distance the fit is quite good. It is not a lens, and the giveaway is the near-and-far split again: fit the polynomial on a target at one metre and it is wrong at three, in a way a real lens never is.

Why the fit is least-squares and not an intersection

A small point about method, because it decides what the reported number means.

The obvious way to ask whether a bundle of rays meets is to intersect them pairwise and see how far apart the intersections are. That is a worse question than it looks. Two rays in three dimensions almost never intersect at all, so the pairwise version has to be softened into “closest approach”, and then the answer depends on which pairs are chosen — a bundle of eleven rays has fifty-five pairs and their closest approaches are spread over a range that includes the answer and a good deal else.

The least-squares common point asks the question the definition actually poses: is there a single point through which all of them pass? It produces one point and one residual, the residual is zero if and only if the bundle really is concurrent, and it degrades gracefully rather than by choosing a favourite pair.

It is also the same instrument the site already uses on the picture side. The vanishing point of a bundle of drawn edges is found by least squares over the image lines, with the residual reported, precisely so that a drawing whose edges do not meet has no vanishing point invented for it. Here the same idea is applied to rays in space instead of lines in the picture, and the same discipline follows: a figure that reported a “centre of projection” for a refracted picture without its residual would be inventing one.

Apparent depth is a curve, not a numberDividing by n is correct looking straight down and nowhere else. At 80° from the vertical the bottom appears at 16% of its true depth rather than 69%.00.2000.4000.600020406080angle of the line of sight from the vertical (degrees)apparent depth, as a fraction of the true depth1/n = 0.690n = 1.45069.0% straight down, 16.3% at 80°
Fig. 5 The caustic, in another coordinate system. Each point of this curve is the depth one ray of the bundle appears to come from, so the spread of the curve is the spread of the fit. Drawn here at an index of 1.45 rather than water’s, which pulls the whole curve down — 69.0% straight down, and 16.3% at 80°.

The scale of the effect, and when to care

The miss in the figure is about ten millimetres, which is the size of a fingernail. Two things scale it.

The first is the standoff — how far the camera sits from the interface. The miss grows roughly in proportion, because a longer air path spreads the rays over a wider range of incidence angles before they reach the water. Pressing the port against the glass of an aquarium is a real mitigation and the reason people do it instinctively.

The second is the field of view. Everything here is a departure that grows with the angle off the optical axis, so a long lens through the same interface is far better behaved than a wide one. That is the opposite of the usual underwater advice, which is to shoot wide to reduce the water between camera and subject — the two considerations pull against each other, and which wins depends on whether the picture is being looked at or measured.

Both scalings have the same source. The bend at the surface is θ−arcsin⁡(sin⁡θ/n)\theta - \arcsin(\sin\theta / n), which is zero on the axis and grows monotonically with the angle. Everything in this field is that function, integrated over whatever path the configuration provides.

The gate this essay forced

There is a piece of machinery that had to be built before this essay could be written, and it is worth stating because it is a piece of the site’s furniture rather than of its geometry.

Every perspective figure here ends with a strip stating the distance the picture is correct from. That strip is the site’s one non-negotiable ornament: a perspective picture is a projection from a point, the point is computable, and stating it is what separates a figure that shows a construction from a figure that makes a claim.

Half the figures in this field cannot print it. A refracted picture has no station point — that is the whole finding — and putting a number there would be inventing one. So those figures print the miss instead, and the site’s gate now asks, of every generator, either for a stated viewing distance or for a declaration that the figure has none, backed by a measurement showing that it has none. A figure cannot simply omit the strip and be quiet about it.

That is a small thing and it is the kind of small thing this fleet keeps finding matters. The failure mode being closed is not a wrong number; it is an absent one, and every gate here asks whether a label fits, contrasts and stays inside the frame, and none of them asks whether it exists.

The same question, asked of a drawing

There is a version of this test that has nothing to do with water, and it is worth stating because it is what makes the field’s instrument general rather than special-purpose.

Hand somebody a hand-made perspective drawing and ask whether it is a projection of anything. The rays are not available — there is no scene and no camera — but the drawn edges are, and bundles of edges that are supposed to be parallel in the world must meet at a point in the picture. Fit each bundle by least squares, report the residual, and the drawing answers the question about itself. That is precisely what the site’s recovery does, and it is why the recovery reports its bundle residuals rather than only its focal length: a drawing whose bundles do not meet has no vanishing points to hand to the recovery, and computing a “least-squares vanishing point” for lines that do not meet would be inventing a number.

So there are two instruments with the same shape, one working in the picture and one in the world. In the picture: do the drawn lines that should be concurrent meet, and by how much do they miss? In the world: do the rays that should be concurrent meet, and by how much do they miss? A hand drawing usually fails the first by a few pixels because the illustrator was working by eye. A refracted photograph passes the first — its lines are near enough straight to fit — and fails the second decisively.

That asymmetry is the useful part. It says the failure of a refracted picture is not visible in the picture. Nothing about the drawn marks announces it, which is why the effect is invisible in every underwater photograph anyone has looked at and shows up immediately in any attempt to measure one.

The rays of a refracted picture, continued into the waterEvery ray leaves the pinhole, bends at the surface and carries on. Fitted to a common point they miss it by 22.0 mm — the circle is that miss drawn at the figure's own scale. With the water removed the same fit misses by 0e+0 m.the water surfacethe pinholethe rays miss by 22.0 mmno single viewpoint — the rays miss by 22.0 mmdry control: 0e+0 m
Fig. 6 The same bundle with the surface twice as far from the eye. The rays span a wider range of incidence angles before they reach it, and the miss grows accordingly — so the failure has a size that depends on the configuration rather than a size full stop.
A case front 25 mm thick, with and without itEvery point has moved — by up to 4.5 px — and every vanishing point has not, to 4e-6 px. So the camera recovered from this picture's own vanishing points is the camera that took it, to 4e-10 relative, out of a picture in which nothing is where it was.no single viewpoint — the rays miss by 4.5 px, depth-dependentf recovered from it: 396.88 px
Fig. 7 The case where the same arithmetic produces the opposite result. With the same medium on both sides of the glass, every direction survives exactly and only the finite points move.

Where this leaves the premise

The site’s premise is that every picture states the point it is correct from. This field contains pictures for which there is no such point, and rather than weaken the premise, that sharpens it.

A picture with a station point is one whose rays meet. That is a property a picture either has or does not, it is testable from the rays alone, and the test has now been run on a case where the answer is no. What the premise says, properly stated, is: when a picture is a projection, the point it is correct from is a computable fact about it and should be stated — and the essays in this field are the ones that had to check the antecedent first.

The next essay is the surprise on the other side. A pane of glass with air on both sides moves every point of the picture and moves no vanishing point at all, so a camera recovered from a picture taken through a shop window is exactly the camera that took it — out of a picture in which nothing is where it was.

Where this goes next

water, n = 1.333eyetruly 1.50 m downappears 0.818 m downh/n would be 1.125 mno single viewpoint — the rays miss by 17.1° of bend at the surfaceapparent depth 54.5% of the true one, not 75.0% What a ray does at a surface A pool looks three-quarters as deep as it is — but only if you look straight down. At sixty degrees the same bottom appears at half its depth, and at eighty at a fifth, which is why the far end of a pool looks shallow enough to walk in. horizonABCDon the groundin the picturelength AB1.00011.3930ratio AB:CD0.56670.6837cross-ratio1.31681.3168correct from 26 cm, at 160 mm wide34° across What a projection destroys A projection loses length, angle, area and the ratio in which a point divides a segment. Exactly one quantity comes through untouched, and almost everything that can be checked about a picture is checked with it — including, as it turns out, some things it cannot check at all. no single viewpoint — the rays miss by 4.5 px, depth-dependentf recovered from it: 396.88 px What survives a pane of glass A slab of glass moves every point of a picture and moves no direction at all. So the camera recovered from a photograph taken through a display case is exactly the camera that took it — out of a picture in which nothing is where it was.