Four numbers and a window

The divide is postponed builds a projection matrix from a camera and two distances and checks that it reproduces this site’s pinhole to five parts in a hundred trillion. This essay asks the question the other way round: given the matrix, which of its numbers are the camera?

The usual presentation makes the question hard to ask. A matrix built from a field of view and an aspect ratio has the answer hidden inside two derived quantities. Built from a general frustum — six numbers, four sides and two planes — it is visible.

Six numbers, and what each of them is for #

The frustum’s six are the left, right, top and bottom sides — measured on the near plane — and the near and far distances themselves. Written that way the matrix has no field of view in it and no aspect ratio, and both of those turn out to be derived quantities that exist only when the frustum is symmetric.

The convention needs one note, because getting it backwards is this site’s oldest recorded gotcha. top is above bottom in a reader’s sense and smaller as a number, because these camera coordinates run y down the page, which is the computer-vision convention this collection uses throughout. A frustum built with the two swapped produces a picture that is perfectly consistent, mirrored vertically, and passes every assertion that asks whether a mark is where it should be relative to other marks.

The four sides are ratios #

The sides are quoted on the near plane, which is what every graphics interface does and what makes them look like distances. They are not distances. Divide any of them by the near plane and it cancels:

$$\frac{\text{left}}{\text{near}} = \frac{-p_x}{f}, \qquad \frac{\text{right}}{\text{near}} = \frac{W - p_x}{f}$$

which are the directions the leftmost and rightmost pixels look along. For the camera in the figure they are −0.4663 and +0.4663, and between them they carry the focal length of 739.85 pixels and the principal point at the middle of a 690-pixel frame, and nothing else.

That is the first claim and it is checkable rather than descriptive. Build the matrix from the six frustum numbers and build it from the camera, and the two agree entry for entry — every one of the sixteen, at every near plane tried, to under a part in a hundred trillion. Move the near plane by a factor of two hundred and the two lateral rows do not change at all.

So an off-centre frustum is not a distorted view. It is the same eye looking through a window that is not centred on it, and the site already has that statement in another vocabulary: the principal point is not the centre is about a shift lens, which moves the principal point without tilting the picture plane. An off-centre frustum is a shift lens written as a matrix.

The two planes are not camera parameters #

Nothing a reader can see moves when the near or far plane moves. Every x and every y in the picture is unchanged, because neither plane appears in the first two rows.

What the planes decide is which points exist — the clipper keeps what is between them — and how the ones that do get ordered, since the third row maps depth into a stored code. Both are real and neither is visible in a still picture of a scene entirely inside the frustum.

That is worth stating plainly because the two planes are usually presented alongside the field of view as though all three were properties of the camera. They are not the same kind of thing at all: a field of view is a fact about an instrument, and a near plane is a decision about a computation. The next two essays are about what each of the planes costs, and the answers are wildly unequal.

What the four sides can be, and cannot #

The sides are four independent numbers, and it is worth asking what arrangements of them are legitimate, because two of the four constraints are not obvious.

Left below right, top above bottom. A frustum with zero or negative width has no picture in it, and the construction refuses rather than producing a matrix that divides by zero somewhere downstream. That refusal matters more than it sounds: a degenerate frustum built from a tile of zero width would otherwise produce a matrix full of infinities, and every point put through it would land at a plausible-looking coordinate.

Near strictly in front of the eye. A near plane at zero is refused. It is the one place in the whole construction where a limit does not exist: the sides are quoted on the near plane, so a near plane at the eye makes all four of them zero and the ratios that carry the camera become nothing over nothing.

The sides need not straddle the axis. This is the one that surprises. Both left and right may be positive, which is a frustum entirely to one side of the optical axis, and the matrix is perfectly well-formed. That is what a tile of the right-hand edge of a picture is, and it is what a projector aimed above its own axis is — a real instrument, drawing a real picture, whose field of view in the usual sense does not contain its own axis.

And near and far are otherwise free. Any positive pair with far beyond near gives a valid matrix. There is no constraint tying them to the scene, and this is exactly why they can be moved without moving a mark.

The bottom row #

The last row of a projection matrix is (0, 0, 1, 0), and it is the only place the depth enters as a divisor. It says: the fourth coordinate of the result is the depth. Everything downstream divides by it, and the divide is what makes the picture a perspective.

Set that row to (0, 0, 0, 1) instead. Now w is 1 for every point, the divide does nothing, and what comes out is an affine map of the scene — a parallel projection, drawn by the same machine through the same window.

It really is one: the flattened matrix preserves midpoints to a few parts in a hundred million million of a pixel, which is the definition parallel projection is not primitive uses, and the same segment through the unflattened matrix misses its own midpoint by nearly thirty-three pixels.

One rescaling is needed when the row is flattened, and it is not a fudge. The lateral entries are ratios that were going to be divided by the depth and now are not, so the drawing has to be told which depth it is true at — the same “one plane is drawn at true size” decision the oblique systems make by decree. That plane is the only parameter the conversion has.

What the flattened matrix is, in the field’s own terms #

Flattening the row does not produce a parallel projection. It produces the orthographic projection along the camera’s own optical axis, at whatever scale the chosen true-depth plane fixes — which is one point of the two-parameter family the axonometric ladder is about, and not a free one.

To get an oblique projection out of the same machine, more than the bottom row has to change. An oblique system’s rays are not perpendicular to the picture plane, and that shows up as off-diagonal entries in the top two rows: a term coupling z into x and into y, which is exactly a shear. So cavalier and cabinet are the flattened matrix with two extra entries filled in, and their kernel is not the optical axis.

Which is the same statement oblique is a shear makes from the drawing-board side, arriving as an entry pattern. The matrix formulation is worth having because it makes the three families’ relationship a matter of which entries are non-zero: a full bottom row is perspective, a flattened one with a diagonal top-left is orthographic, and a flattened one with off-diagonal terms is oblique. Three fields of this site, distinguished by the sparsity pattern of one four-by-four array.

Which makes the limit a matrix entry #

The eye taken to infinity says a parallel projection is a photograph from infinitely far away with the lens lengthened to match, and watches the limit happen. This is the same statement with the limit already taken.

The two accounts have to agree, and the agreement is measurable. Move a real pinhole back to distance D with its focal length grown as D·k so the subject stays the same size, and difference its picture against the flattened matrix’s. The departure falls as D to the power 1.007 — one over the distance, fitted rather than assumed.

An exponent of one rather than two is worth a sentence. The leading term in the departure is the scene’s depth range divided by the eye’s distance, so halving the depth of the object is exactly as good as doubling the distance — which is why a long lens flattens a shallow subject convincingly and a deep one not at all.

Fitting that exponent turned up a trap the collection’s convergence gate now guards. Measured against a reference camera four times beyond the last rung of the ladder the exponent came out 1.13; measured against one four orders beyond it, 1.01. The reference was itself a perspective picture, and its own error was being subtracted from every rung. A limit measured against a nearby stand-in is a limit of the stand-in.

The window metaphor, taken literally #

The word window is doing real work here and not decoration, so it is worth pushing until it breaks.

A person at a fixed place in a room, looking through a rectangular hole in a wall, sees a picture. The four sides of the hole are the four sides of the frustum: they decide what is in view, and moving the person changes the picture while moving the hole changes only the framing. That is the sense in which the four sides carry the principal point — an off-centre hole is a hole with the eye not opposite its middle, and the picture through it is a crop of the picture that a centred hole would give, not a different projection.

The near and far planes have no counterpart in the room, and that is the honest limit of the metaphor. Nothing in a wall says stop drawing things closer than ninety centimetres. They exist because the computation needs somewhere to put the depth: a stored depth has to be a bounded quantity mapped into a fixed range of codes, and the two planes are what fix the range.

So the metaphor is exact for the four numbers that are the camera and empty for the two that are not, which is a reasonable way to remember which is which. Anything a person leaning through a window could change is in the sides; anything only a computation cares about is in the planes.

So the accounting is #

Six numbers and one row, and they divide into three kinds:

Four sides — the camera. Focal length and principal point, and recoverable from a picture: this site takes cameras out of pictures throughout the foundations field, and what comes back is exactly these.

Two planes — a window on depth. Not recoverable from a picture, because nothing in the picture depends on them. Choices about what the computation keeps and how well it orders it.

One row — whether it is a perspective at all. Two values, and the choice between them is the choice between the construction field and the parallel field, made in one matrix entry.

Reading the matrix back #

Because the four sides carry the camera and nothing else, the camera can be read out of a matrix by inverting one line of arithmetic — which is a small thing and is the reason this decomposition is worth having at all.

From the first row’s two entries, 2·near/(right − left) and −(right + left)/(right − left), the focal length is the first times the frame’s half-width, and the principal point is where the second says the axis crosses. The second row gives the same two numbers again, from the height, and the two answers must agree for square pixels; where they do not, the difference is a pixel aspect, which is exactly the quantity the pixel that is not square is about and which an unmodelled version of absorbs into a confident wrong camera.

So a projection matrix is a camera plus two window parameters, and the separation is not a matter of interpretation — the entries are in different rows. That is the whole reason this essay can make a claim about which numbers are the camera rather than an argument about which ones feel like it.

There is one thing the reading cannot recover and it is worth naming. Nothing in the matrix says where the eye is or which way it is pointing; those live in the view transform that runs before it, and the projection matrix is written in the camera’s own coordinates throughout. So “the camera” here means the intrinsic camera — focal length, principal point, pixel shape — and the extrinsic half is somewhere else entirely. The site’s own recoveries take both out of a picture, and they take them out by different routes for the same reason.

What this says about the field of view #

The tidiest consequence is a small correction to how a rendered camera is usually described.

A field of view is not one of the six numbers. It is a derived quantity: for a symmetric frustum it is 2·arctan((right − left) / 2·near), and it exists only because the frustum happened to be symmetric. Give the frustum an off-centre principal point and there is no single field of view — there is a left half-angle and a right half-angle, and they differ.

That is not a defect of the description; it is the description catching up with the instrument. A shift lens has exactly that asymmetry, a tilted projector has it, and every tile of a tiled render has it. The essays that follow are all cases where the symmetric description has nothing to say and the six numbers do.

And it explains why a focal length is not an angle needs a rectangle named before it can quote a field of view. A focal length lives in the four sides; an angle needs the sides and the width they are measured across, and the width is the thing the sensor supplies. The frustum contains both, which is why it needs six numbers where a focal length needs one.