What happens behind the eye

This site’s camera has refused to project a point behind the eye since the first commit, and the reason is given beside it in one sentence: returning nothing is “a refusal rather than a wrapped-around point appearing somewhere plausible in the picture.”

This essay is about the phrase somewhere plausible, because it is the whole difficulty and it is almost always described in a way that undersells it. The standard account says that points behind the camera “project incorrectly” or “must be clipped away”. Both are true and neither conveys what is actually drawn, which is a completely convincing picture of something that is not there.

The arithmetic, which takes one line #

A world point becomes four clip coordinates $(x_c, y_c, z_c, w)$, and $w$ is the depth along the optical axis. Positive in front of the eye, zero on the eye plane, negative behind it.

The picture coordinates are $x_c/w$ and $y_c/w$. If $w$ is negative, both quotients change sign, and a change of sign in both picture coordinates is a rotation of 180° about the principal point.

So a point 3 m behind the eye and 2 m to the left is drawn as if it were 3 m in front and 2 m to the right, at a scale that gets it roughly right. It is not off the edge of the frame. It is not at infinity. It is not NaN. It is a point, in the picture, in a believable place.

Why the failure is worse than it looks #

Three properties combine badly, and it is worth separating them because each one alone would be manageable.

A single point behind the eye is invisible as an error. There is nothing wrong with its image considered on its own. It is a well-formed pair of coordinates in the frame.

A segment with one end behind is drawn reversed. This is the measurable one. Take a segment from 2.2 m in front to 3.6 m behind. Clipped, it runs from its front end to the near plane and stops. Unclipped, its far end lands on the opposite side of the principal point, so the drawn line runs away from where the segment goes. The direction cosine between the two drawings is −0.9999999999999998, which is as close to exactly −1 as a floating-point computation gets.

That number is the measurement this essay is built on, and the reason it is a cosine rather than a distance is worth stating. A displacement in pixels can always be argued about — is 8,297 px a lot on a 690 px frame? Obviously, but the question invites a threshold. A direction cosine of −1 admits no threshold at all: the line is not in the wrong place, it is pointing the other way.

And a whole surface half behind the eye is drawn as a plausible surface. This is the failure that ships. A wall the camera is standing against, a floor extending under the viewer’s feet, a ceiling passing overhead — each of these is a large flat thing whose near part is behind the eye, and the unclipped drawing of one is a large flat thing at a strange angle. There is no ragged edge to notice, no obvious infinity, no missing geometry. There is a wall where no wall is.

Clipping is not a rendering optimisation #

The commonest description of clipping is that it saves work: no point drawing what will not be seen. That is a real benefit and it is not the reason clipping exists.

Clipping exists because the divide is undefined at $w = 0$ and wrong for $w < 0$, and a clipper is the only thing that guarantees the divide is never asked to do either. Every other benefit — not rasterising off-screen triangles, not shading invisible fragments — is a bonus attached to a step that would be mandatory if it saved nothing at all.

That is why the operation is placed where it is. In clip space, before the divide, both endpoints of a segment are ordinary finite numbers with an honest sign, and the crossing point is the root of a linear function:

$$t = \frac{d_a}{d_a - d_b}, \qquad d = z_c + w$$

where $d$ is the signed distance from the near plane, which in clip coordinates is the plane $z_c = -w$. One subtraction and one division, and the segment has been cut exactly where it crosses.

After the divide, none of that is available. One endpoint has already become a reflected point of the wrong sign, so interpolating between the two endpoints interpolates between a real position and a fictitious one and the result is not the crossing.

Two refusals, one fact #

This site’s pinhole and the pipeline’s clipper are two different pieces of code written for two different purposes, and they must agree about which points have no image at all. The field’s gate asserts that they do: the point the clipper removes is the point the site’s own camera refuses to project, and a segment wholly behind the near plane is refused by the clipper rather than shrunk to nothing.

That agreement is not automatic and it is worth having as a check rather than as an assumption. A clipper that quietly returned a zero-length segment instead of nothing would produce an empty drawing with every assertion passing — the failure mode kept() exists in the shared kit to catch, and which the curved field met when a wrong clip box left a figure with an empty frame and a full set of green checks.

What is really being said about a projection #

There is a statement about projective geometry underneath all this and it is worth making carefully, because the sloppy version of it is what makes people think clipping is a technicality.

The sloppy version says: in projective geometry a point and its antipode are the same point, so the “reversal” is not an error at all, merely a consequence of working in a space where sign does not exist. That is a correct description of the projective plane and a wrong description of what a camera does.

A camera is not a map from the projective plane. It is a map from the half-space in front of the eye. The projective plane identifies a direction with its opposite; a camera absolutely does not, because one of those two directions has light coming from it and the other has the back of the camera. The picture surface is a plane, not a projective plane, and the eye is on one side of it.

So the cleanest statement of what the clipper does is: it enforces the fact that this site has stated in every field, which is that a picture is a projection from a point of the things on one side of that point. Anamorphosis is a projection from a point; a shadow is a projection from a lamp; a reflection is a projection from a camera behind the mirror. In every one of them, the centre has a front and a back, and the machinery is only correct on the front.

The case the sign test does not catch #

There is a tempting shortcut: rather than clip, test each point’s $w$ and discard the ones that are negative. It is one comparison per vertex and it removes the points that would be drawn wrongly.

It is also not sufficient, and the reason is exactly the reason clipping is a segment operation rather than a point one. A segment with one endpoint in front and one behind has no vertex that fails the test in a way that fixes the segment. Discard the bad endpoint and the segment vanishes, taking with it the part that really is in front of the camera and really should be drawn. Keep it and the line is reversed. Neither is right, because the correct answer is a new endpoint that was not in the original data — the point where the segment crosses the near plane.

This is the same shape as a finding the refraction field recorded about total internal reflection: a test that correctly identifies which inputs are bad does not thereby produce the right output for the ones that are partly bad. Being able to detect a failure and being able to repair it are different capabilities, and conflating them is how a check that passes ends up shipping a wrong picture.

Why the near plane cannot be at zero #

The obvious question, once the near plane is understood as the clipping plane, is why it is not simply placed at the eye — clip exactly at $w = 0$ and nothing further is thrown away.

Two reasons, and only one of them is about numbers.

The arithmetic reason is that the depth row of the projection matrix is $A + B/z$, and $B = -2fn/(f-n)$ goes to zero as the near plane does. A near plane at zero gives a depth map with no resolution anywhere; a near plane very close to zero gives one whose resolution is spent entirely on the first few centimetres. That is the whole subject of this field’s next essay, and it means the near plane is not free: pushing it toward the eye to avoid cutting things buys the cut and pays for it in depth precision everywhere else.

The geometric reason is that a plane at the eye is not a plane the projection is defined on. Points on the eye plane have $w = 0$ exactly and no image at all — they are at infinity in the picture, in the same sense a vanishing point is. Clipping at the eye plane leaves points arbitrarily close to it, whose images are arbitrarily far outside the frame, and a renderer that has to handle coordinates of $10^{12}$ px is a renderer that will overflow something.

So the near plane is a compromise with two independent pressures on it, and there is no value that satisfies both. That is worth saying because “set the near plane as small as possible” is common advice and it is wrong in a way that shows up nowhere near where the setting was made.

What this looks like when it goes wrong in practice #

Three symptoms, all of which are the same fact and none of which look like each other, which is why this is worth spelling out.

Geometry that appears when the camera moves close to a wall. The wall’s far part is in front, its near part is behind, and the unclipped drawing folds it through the principal point.

A shadow, reflection or projected texture that inverts. All three are projections from a centre — the site has made that argument three times — and all three have a half-space behind their centre. A shadow computed without checking which side of the lamp the caster is on produces a shadow on the wrong side, at the right size, with the right shape.

A rectangle drawn as a bow-tie. When two of four corners cross the eye plane, the quadrilateral’s corners come back in an order that crosses itself. This site shipped exactly that, from a different cause — a shadow polygon whose vertex list was reordered by Array.prototype.filter — and the light field records it. Two unrelated bugs with an identical symptom is a good reason to distrust the symptom and check the mechanism.

The other four planes, and why only one of them is compulsory #

A frustum has six faces and a clipper is usually described as cutting against all of them. It is worth separating the one that is mandatory from the five that are not, because the distinction says what clipping is for.

The near plane is compulsory. Everything above: the divide is undefined at $w = 0$ and wrong beyond it, and no other step can repair a point that has already been divided by a negative number.

The far plane is compulsory only because the depth row exists. A pinhole has no far plane; a matrix with a depth row does, because $A$ and $B$ are fixed by naming two distances and a point beyond the far one produces a depth code outside the storable range. Drop the depth row and the far plane goes with it.

The four side planes are optional. A triangle half outside the frame can simply be drawn, with the parts outside the picture discarded pixel by pixel as they are written. That is slower and it is not wrong, and plenty of renderers do exactly that rather than cutting geometry against the sides. The reason the sides are usually clipped anyway is arithmetic hygiene: a triangle extending a million pixels beyond the frame is a triangle whose edge stepping has to stay accurate over a million pixels, and it is cheaper to cut it than to be careful.

So of six planes, one is a correctness requirement, one is an artefact of how depth is stored, and four are an optimisation. Textbooks present the six as a set and the set has three different justifications in it.

The one-line summary, and the reason it is not enough #

Clip before the divide.

Every graphics text says it, and the reason to spend an essay on it is that the sentence is usually offered as a rule and almost never as a fact about pictures. The fact is that the fourth coordinate is the only thing in the pipeline that knows which side of the eye a point is on, that the divide consumes it, and that the drawn result of consuming it is not an error a reader can see.

Which is the same reason this site prints a viewing distance on every figure. A picture does not carry the conditions under which it is a picture of anything; those have to be stated separately, and the ones that are not stated are the ones that go wrong quietly.