The two points a picture hides

Two phases ago this site built a ladder. A photograph of a plane determines the plane projectively; adding the vanishing line makes it affine, so equal divisions and midpoints come back; adding one more fact makes it metric, so angles and ratios of lengths come back; and no amount of further looking supplies a length.

The rungs are right and the third one is vague. One more fact covers an imaged circle, a pair of known-perpendicular directions, a known aspect ratio, a calibrated camera and several other things, and they cannot all be the same fact. What is the quantity being bought?

It is two points, and this essay is about what they are, how to get them, and exactly what they do and do not buy.

The upgrade, counted #

The counting is the quickest way to see that the answer has to be two points.

A projectivity of the plane has eight degrees of freedom: a 3×3 matrix up to scale. An affinity has six: the last row is fixed at $(0, 0, 1)$. A similarity has four: rotation, uniform scale, and two translations.

So the projective-to-affine step costs two numbers, and the affine-to-metric step costs another two. Four numbers in total, arriving in two instalments of two.

Two real numbers is a point on a line. And four numbers, as two points constrained to lie on a known line, is exactly the affine and metric steps taken together — which is what the circular points are: two points, at $(1, \pm i, 0)$, on the line at infinity.

The instalments are the two ways of reading the same purchase. Knowing the line the two points lie on is the affine rung, and it is two numbers. Knowing where on it they are is the metric rung, and it is two more.

What they are, in one sentence #

Every circle in a plane meets the line at infinity at $(1, i, 0)$ and $(1, -i, 0)$, whatever its centre and radius. Every similarity of the plane fixes those two points, and no other projectivity does.

The second half is why they matter. A projectivity of a plane is a similarity if and only if it fixes the circular points, so knowing where they are is knowing which maps are similarities — which is what it means to know the plane’s metric structure up to scale.

That is a definition rather than a tool, and the tool comes from applying a projection to it. The images of the two points under a projection are two points in the picture, and a homography that carries those to the standard pair carries the picture back to a similarity of the plane.

Getting them out of a picture #

They are where the plane’s vanishing line meets the image of any circle in the plane. Both halves of that are inherited: the points lie on the line at infinity, so their images lie on its image; and they lie on every circle, so their images lie on the image of any one.

A line meets a conic in two points, so the intersection hands back exactly the pair, with nothing fitted and nothing chosen.

The intersection is complex in the ordinary case. A circle wholly in front of the camera has an image that does not reach the horizon — the discriminant here is $-1.3\times10^2$ — and a routine returning “no intersection” would be throwing the answer away rather than reporting an absence. The first coordinate of the pair here is $345.0 - 909.3i$, and it is worth noticing that the real part is exactly the principal point’s own $x$: the horizon in this camera is horizontal, so the pair’s real part is where the optical axis crosses it.

What the discriminant’s sign is a statement about #

The discriminant is not merely a flag for which branch of the code to take; it says something about the scene, and saying what makes the complex case less strange.

The horizon is the image of the plane’s line at infinity, so it separates the images of points in front of the camera from the images of points behind it. A circle lying wholly in front therefore has an image lying wholly on one side of the horizon, the line misses the conic, and the two intersections are complex. A real pair means the imaged circle crosses the horizon, which means the circle passes through the plane’s own line at infinity — that is, it is not a bounded circle in front of the camera at all.

So the complex case is not the awkward case. It is the only case a photograph of an ordinary circle can produce, and a real discriminant is evidence that something is wrong with the input rather than that a different branch is needed.

The zero case — the horizon tangent to the imaged conic, the two points coinciding, the upgrade collapsing because a repeated pair carries one number instead of two — is worth naming and cannot arise from a genuine circle. No circle’s image ever reaches the horizon at any radius, for the reason two lines at infinity sets out: a circle has no point at infinity in its own plane, so its image cannot touch the image of that plane’s infinity. A discriminant of zero, like a positive one, is evidence about the input rather than a branch to handle.

Which gives a free check the construction does not use. Any second circle in the same plane must return the same two points, since they belong to the plane rather than to the circle. So a picture with two circles in one plane over-determines the pair by two numbers, and the disagreement is a residual — two circles, one picture is where this collection runs that comparison, and it is a test on the picture rather than an improvement to the answer, exactly as a fifth point is for a conic through five.

The refusal the intersection needs #

The complex case is not an edge case here; it is the ordinary one, and a routine that treated it as a failure would fail on almost every picture.

A conic and a line meet in two points. When the line cuts the visible conic those two are real and can be pointed at. When it misses — which is what happens whenever the imaged circle lies wholly below the horizon, which is whenever the circle is wholly in front of the camera — the two points are a conjugate pair, and they are just as much intersections as the real ones.

So the machinery returns them, with the discriminant beside them so the caller can tell which case it is in, and the site’s gate asserts the ordinary case comes back complex rather than empty. The assertion is worth having because the natural implementation of “where does this line meet this conic” returns an empty list for a negative discriminant, and an empty list here is the metric upgrade silently declining to happen.

The real case is not a pathology either. It occurs when the imaged circle reaches the horizon, which happens when the world circle crosses the plane through the eye — part of it is behind the camera, its image is a hyperbola, and the two branches run to the horizon in two real directions. Those two real points are still the imaged circular points, and everything below works unchanged.

The dual conic, and why the detour #

Working with the two points directly is awkward, because building a rectifying homography out of a complex conjugate pair means keeping track of complex scale factors that have to cancel.

There is a real object carrying the same information:

$$\mathbf{C}^*_\infty = \mathbf{I}\mathbf{J}^{\mathsf T} + \mathbf{J}\mathbf{I}^{\mathsf T}$$

It is real for any conjugate pair — the imaginary parts cancel term by term, and the machinery asserts that rather than assuming it — symmetric, and rank two. In the plane’s own coordinates it is $\mathrm{diag}(1, 1, 0)$ up to scale, which is the cleanest possible statement of what the pair is: the degenerate conic consisting of those two points.

The rectification then falls out of a symmetric eigen-decomposition. Diagonalise the imaged $\mathbf{C}^*$, and the map sending it to $\mathrm{diag}(1, 1, 0)$ is $\Lambda^{-1/2}\mathbf{U}^{\mathsf T}$ on the two nonzero eigenvalues. Twelve lines of arithmetic and no complex numbers after the first step.

The vanishing line comes out on the way #

There is a detail in that decomposition worth stopping on, because it makes the counting above visible rather than argued.

The dual conic is rank two, so it has a null eigenvector — and that eigenvector is the vanishing line. It is not supplied to the decomposition and it is not fitted; it falls out.

Which is the four-numbers-in-two-instalments statement arriving as arithmetic. Two points on a line determine the line, so a construction given the two points has been given the line as well, and the affine rung is bought on the way past the metric one without a separate purchase.

The check is direct: the null eigenvector’s alignment with the horizon computed independently from two ground directions is 1.000000000000.

What comes back, and what does not #

The rectification is run on a test rectangle the machinery was never shown, and three things are asserted.

Every angle comes back, at a worst departure of 2.2e-13 degrees over every ordered triple of five test points.

Every ratio of lengths comes back, at a worst relative departure of 5.6e-15 over every pair of pairs.

No length comes back, and this is asserted by being refused rather than by being small. The rectified plane is a similarity of the real one; its overall scale is a free parameter, and the figure prints a dash where a length would go.

That dash is a convention this site adopted two phases ago and it is load-bearing. A number in that cell — the rectified rectangle happens to come out 3.4 units wide, say — would be read as a measurement of the world, and it is a measurement of the decomposition’s arbitrary normalisation.

Why exactly this and not more #

The natural question is whether a cleverer construction could get the length too, and the answer is a short one.

A projection of a plane is a homography, and a homography composed with a uniform scaling of the world plane is a different homography that produces the identical picture — because scaling the world and moving the camera in proportion cancel. So two scenes differing by a scale factor are photographically indistinguishable, which the metrology field demonstrates by projecting a world $137\times$ larger from $137\times$ further away and getting the same picture to arithmetic noise.

There is nothing in the picture to operate on. The missing length is not hidden, it is absent, and the only cure is to import one from outside — a measured distance, a known object, a printed sheet.

Where the conditioning lives #

The two numbers that place the pair along the vanishing line are the metric rung, and they are not equally easy to come by. It is worth saying where the difficulty sits, because the failure is quiet.

The pair’s position depends on the fitted conic’s coefficients near the vanishing line, which is where the imaged circle is not. A circle photographed obliquely occupies a band well below the horizon, so the intersection is an extrapolation of the fit rather than an interpolation of it — and an extrapolated conic is the least reliable thing a conic fit produces.

The practical consequence: the pair is well determined when the imaged circle is large and strongly foreshortened, so that its own long axis points at the horizon and the fit has leverage there, and poorly determined when the circle is small or nearly frontal. A nearly frontal circle images as a nearly circular ellipse whose intersections with a distant horizon are set by coefficients the samples barely constrain.

None of that shows up as a failure. The rectification is produced, it is a valid similarity of something, and its angles are self-consistent. What has moved is which plane it is a rectification of. That is the same shape of failure the lens field records for a distortion coefficient fitted from a short arc, and the same repair applies: report the fit’s own conditioning beside the answer rather than the answer alone.

Five references, one purchase #

The vagueness in “one more fact” resolves once the purchase is named, because every route to the metric rung is a route to the same two points.

An imaged circle gives them directly, as the horizon’s intersection with it.

Two known-perpendicular directions give a constraint each on the pair’s position along the vanishing line; two such pairs pin it.

A known aspect ratio of an imaged rectangle does the same thing in one step, which is how the facade essay proceeds.

A known angle between two imaged lines is Laguerre’s formula run backwards: the angle and the two lines determine the cross-ratio, which determines the pair.

A calibrated camera gives them for every plane in the picture at once, and that is a strong enough statement to deserve its own rung.

Five references, one thing bought. That is what makes the two points the right way to describe the metric upgrade: it is the invariant description, and the references are five ways of paying for it.

What the pair does not do #

Three limits, and they follow from what the pair is rather than from any weakness in the method.

They belong to one plane. Every plane in a picture has its own vanishing line and its own pair on it. Rectifying the ground does nothing for a wall.

They are only as good as the conic. The pair is read off a fitted conic and a fitted horizon, and both come from measurements on the picture. A circle sampled over a small arc, or a horizon fitted from nearly-parallel image lines, gives a pair whose position along the vanishing line is poorly determined — and the metric rung degrades smoothly rather than failing.

They buy a similarity, not a rigid motion. Angles and ratios return; the handedness and the orientation of the rectified copy are as arbitrary as its scale, and a rectification that flipped the plane would satisfy every assertion here.

What this makes of the ladder #

The stratification rung can now be restated in a form that says what is being bought rather than listing what will pay for it.

A picture of a plane is a picture of the plane’s projective structure. Everything a cross-ratio can see is there and nothing else is.

The affine structure is one line — the vanishing line — and having it makes parallelism, midpoints, equal divisions and ratios along a line available.

The metric structure is two points on that line, and having them makes angle and ratios across directions available.

The scale is not in the picture at all, and no number of further points helps.

Written that way the ladder is a statement about how much of the plane’s structure the picture has thrown away, in units of points and lines rather than in units of what can be measured. Eight degrees of freedom in the projectivity; two spent to get to affine; two more to get to a similarity; four left, which are the similarity itself and are not a loss.

That accounting is what makes the third rung’s vagueness disappear. It was vague because it was named by its receipts.

The short version #

The metric upgrade of a photographed plane costs four numbers, and they are two points: the images of the circular points, complex conjugates lying on the plane’s vanishing line.

Knowing the line they lie on is the affine rung. Knowing where on it they sit is the metric rung. A construction handed the two points gets the line for free, which the eigen-decomposition demonstrates by returning it as its own null vector.

A rectification built from the pair and nothing else recovers every angle to 2.2e-13 degrees and every ratio of lengths to 5.6e-15, and prints a dash for the length — which is not a limitation of the method but a fact about single views, and the same fact this site has recorded in four fields.