The pixel that is not square

Everything on this site so far has assumed one focal length. The site’s camera takes a single focal length, the recovery returns a single number, and the round trip compares one against one. That assumption is so uniform that it has never been stated as an assumption.

It is one, and it is false of several real cameras. Anamorphic cinema optics squeeze the image horizontally on the way onto the film and unsqueeze it on the way out. A sensor whose photosites are wider than they are tall does the same with no optics at all. Several video formats have carried non-square pixels since they were defined.

Every one of those is a camera with two focal lengths — $f_x \ne f_y$ — and it is a perfectly ordinary projective camera. The picture it makes is not distorted in any sense this site uses the word: straight lines are straight, cross-ratios survive, vanishing points converge exactly. It is a pinhole picture with one axis scaled.

What the recovery has to work with #

The site’s round trip is three equations. Three mutually orthogonal world directions have three vanishing points; each pair of them gives one equation of the form $(v_i - p)\cdot(v_j - p) = -f^2$; and the three together determine the principal point $p$ (as the orthocentre of the triangle) and the focal length $f$.

Three equations, three unknowns — $p_x$, $p_y$, $f$ — and the system is exactly determined.

A camera with two focal lengths has four unknowns. Three equations cannot find four numbers, so something has to give, and the whole interest of this essay is in which thing gives and what the machinery says about it while it happens.

What actually gives #

Run the box through a camera with a squeeze of 2 and hand the drawn edges to the recovery:

The horizontal component of the principal point is exactly right, because the squeeze was about it. The vertical component has moved 158 px. The focal length is roughly halved, though not exactly — the squeeze is absorbed by the focal length and the principal point together, in a proportion the geometry decides rather than by a rule anybody would guess.

And the two quantities the site relies on to report trouble both report nothing.

The two alarms, and why they are silent #

This is the part worth taking slowly, because both alarms were built deliberately and each is a good alarm.

The spread. recoverCamera computes the focal length three times, once from each pair of vanishing points, and returns their spread rather than averaging them away. Its own comment says why: “a construction that is not a projection of a rectangular box gives three different answers, and the spread is how far from being a box it is.” That is the check the wrong field is built on — it is what says a hand-drawn cube is not one.

Here the spread is $0.0 \times 10^{0}$. The three estimates are identical.

The residual. The vanishing-point fit takes a bundle of drawn segments by least squares and returns the worst distance from the fitted point to any of the lines. A bundle that is not really concurrent has a large residual, and that residual is what separates a real projection from a plausible drawing.

Here it is $6.3 \times 10^{-13}$ px. The edges really are concurrent.

Neither is malfunctioning. The squeezed picture genuinely is a perfect projection of a rectangular box — by a camera with square pixels, a focal length of 453 px and a principal point 158 px above where the renderer put it. That camera exists, it is consistent, and it reproduces every drawn mark exactly. The recovery found it and was right about it.

The shape this phase keeps meeting #

The recovering phase’s own recurring finding was that four distinct failure modes in multi-view geometry all return a confident wrong answer with a small residual. This is the same shape, in one view, and it is the sharpest instance the site has.

It is worth naming exactly what makes it sharp. In the multi-view cases the residual is small; here it is at the noise floor. In those cases some diagnostic usually flickers; here every diagnostic the site owns is not merely quiet but at its theoretical best. There is no number to look at, anywhere in the machinery, that distinguishes the two cases.

The reason is structural rather than unlucky. A fit with fewer equations than unknowns does not produce a residual — it produces a family of solutions and picks one, and the picked one fits perfectly by construction. Residuals report over-determination failing; they say nothing about under-determination. That is a general fact about fitting and it is worth stating plainly, because “the residual was small so the answer was right” is an inference people make constantly and it is only valid when there were spare equations.

What would detect it #

Three things would, and it is worth saying what each needs, because the list is short and every item costs something the single view does not have.

A fourth constraint. A fourth direction known to be orthogonal to the others is impossible in three dimensions, but a known angle between two directions, or a known length ratio in the scene, adds an equation and makes the system determined. That is a scene assumption — the picture does not supply it.

A circle. A circle in a world plane images as an ellipse under a square-pixel camera, and the ellipse’s axes are constrained; under a squeeze the constraint is violated in a way that is detectable. This is close to what the foundations field measures about a projected circle, and it needs the scene to contain a circle known to be one.

A second view. Two views of a rigid scene over-determine the intrinsics, and the squeeze then shows up in the residual. Which is what a second picture buys, in its most concrete form: not shape, not size, but the ability of a fit to fail.

Each of those is the same purchase in a different currency. What is missing is not precision — the numbers here are exact to the last bit — it is redundancy, and redundancy is the only thing that lets a wrong model announce itself.

Why the squeeze lands where it does #

The table above says the focal length is roughly halved and the principal point moves 158 px vertically. Neither is obvious in advance and it is worth working out where they come from, because the split between the two is the interesting part.

A horizontal squeeze by $a$ about the picture’s vertical centre line moves every vanishing point toward that line by the factor $a$ and leaves its height alone. So the triangle of three vanishing points is compressed horizontally and not vertically.

The principal point is the orthocentre of that triangle. An orthocentre does not transform with the triangle under a non-uniform scaling — that is precisely what “non-uniform” costs — so it moves, and it moves in the direction the squeeze did not act. Hence a vertical displacement from a horizontal squeeze, which reads backwards until the mechanism is written down.

The focal length then comes out of the dot products taken about the moved orthocentre, so it absorbs both the compression and the displacement together. That is why 898.8 goes to 453.0 rather than to exactly half of 898.8: the squeeze is 2, but part of it has been spent moving the principal point.

The split between the two is decided by the triangle’s shape, which is decided by the camera’s orientation relative to the box. So the same squeeze on the same camera pointed differently gives a different split — a different focal length and a different principal-point displacement, both wrong, both self-consistent. There is no fixed correction factor to be had, which is the practical form of the whole finding.

The convention that makes this benign in practice #

A note in the other direction, because the essay would be misleading without it.

In practice a pixel aspect ratio is declared. Every camera calibration in real use fits $f_x$ and $f_y$ separately from a target with many known points, which supplies far more equations than unknowns and determines both. Video formats with non-square pixels carry the ratio in their metadata. Anamorphic optics have a stated squeeze factor and the unsqueeze is applied before anything else looks at the picture.

So the failure this essay measures is not one that happens to careful people using calibrated equipment. It happens when a picture arrives without its metadata, when a still is grabbed from a stretched video, when an image is resized non-uniformly somewhere in a pipeline, or when a measurement is made from a frame nobody calibrated.

Which is exactly the situation single-view metrology is usually in. Reading a height from a photograph is a technique for pictures that were not taken for the purpose, by cameras nobody has, under conditions nobody recorded — and it is the situation in which the assumption of square pixels is both necessary and unverifiable.

The fifth parameter nobody fits #

There is a fifth intrinsic parameter that this essay’s arithmetic quietly assumes away, and it is worth naming because it is the one place the analogy with real calibration breaks.

A general intrinsic matrix has a skew term as well as two focal lengths and two principal-point coordinates — the entry that appears when the sensor’s rows and columns are not perpendicular. Five parameters, and the three-equation recovery is short by two rather than by one.

In practice skew is fitted as zero by essentially everybody, and for a good reason: a sensor whose photosite grid is not rectangular is a manufacturing defect rather than a design, and measured values come out at $10^{-4}$ of the focal length or smaller. It is the one intrinsic parameter that is genuinely negligible.

But its being negligible is a fact about how sensors are made, not a consequence of anything geometric — and it is worth saying so, because “skew is zero” is usually presented as though it were a property of cameras rather than of factories. An anamorphic lens whose squeeze axis is not aligned with the sensor’s rows produces exactly a skew, and it is not small.

So the honest count is: five intrinsic parameters, three equations from one view of a box, and two assumptions doing the work. This essay measures what happens when one of the two is wrong. The other, being wrong far less often, has never been tested here — which is a shortfall recorded rather than a gap papered over.

The one cue that would give it away #

Having said that no quantity computable from the picture distinguishes the two cameras, it is worth conceding the one thing that does — because the concession sharpens what the claim actually is.

A viewer looking at a squeezed picture of a familiar object sees a squeezed familiar object. Faces are the extreme case: a 5% horizontal squeeze on a face is noticeable to almost anybody, which is a far finer discrimination than any of this essay’s numbers.

That is not a counter-example to the claim. The viewer is supplying an external fact — that faces have a known shape — which is exactly the fourth constraint the recovery lacked. Every one of the three detectors listed above is a way of supplying such a fact mechanically, and a viewer’s familiarity with faces is the same purchase made by a route the machinery does not have.

Which is the site’s usual boundary, met from an unusual side. The geometry says what the picture contains; a viewer regularly knows more than the picture contains, and the extra is where the discrimination comes from. It is worth being precise that the extra is knowledge about the scene rather than sharper looking at the picture.

The honest statement of the limit #

Stated as carefully as the site can:

A single view of a rectangular box, with no scene knowledge beyond the orthogonality of its three edge directions, determines three camera parameters and cannot determine four. If the camera has four, the recovery returns a three-parameter camera that reproduces the picture exactly, and no quantity computable from the picture distinguishes the two.

That is a statement about information rather than about method. A better solver does not help. More careful drawing does not help. Any procedure whatever, applied to that picture, faces the same shortfall, because the shortfall is in the picture.

It is the same form as the site’s oldest limit — one view gives ratios and no size — and it is worth adding to the list rather than treating as a special case. Both are statements that a picture is a record of a pencil of rays, and that anything not determined by that pencil is not in there to be found.