The two pencils keep one number

A point is a line over there found that a mark in one photograph fixes a line in the other, that all such lines pass through the image of the other eye, and ended by describing the whole structure as a book: every plane through both eyes is a page, the line between the eyes is the spine, and each page cuts each picture in one line. The lines in the first picture form a pencil through its epipole; the lines in the second form a pencil through the other epipole; and the pages pair them.

That pairing was stated and not measured. It is a map from one pencil of lines to another, and a map between two sets that are each one-dimensional — a line of lines, parametrised by which page they belong to — has a specific and very restricted form. Measuring it says exactly what the two pictures agree on about the pages, and it turns out to be one number for every four lines and nothing else.

The angles do not survive #

The pair is deliberately unlike the courtyard’s standing pair. The second camera has stepped forward and to the right, is aimed well to the left of the courtyard’s middle and carries a wider lens, so that it sees the pages of the book from a very different direction from the first camera. Both epipoles are finite points, the first inside the left picture and the second just beyond the right edge of the right one, so both pencils visibly converge.

Four marks are chosen spread round the left epipole, and each gives a line in the left picture — through the epipole and the mark — and its epipolar line in the right picture. The angles between neighbouring lines are 46.4°, 15.3° and 13.1° on the left. The corresponding angles on the right are 35.0°, 11.7° and 9.7°.

The largest difference is 11.4°. The map from one pencil to the other does not keep angles between lines, which is no surprise: angles are not kept by any projection, and each picture’s view of the pages is a projection of them.

A pair built less carefully would have hidden that. The first arrangement tried for this essay had the second camera aimed at the courtyard’s middle like the first, with a narrower lens, and its angles were 53.6°, 72.1° and 33.3° on the left against 53.9°, 71.7° and 33.3° on the right — agreement to within half a degree. Half a degree of agreement is not a property of the map, and the reason it happened is worth having, because it is the key to the whole figure; it comes two sections below.

What does survive #

The pencil map keeps one number built from four lines: their cross-ratio.

For four lines through a common point, the cross-ratio can be computed from the sines of the angles between them, $\frac{\sin\theta_{13}\,\sin\theta_{24}}{\sin\theta_{23}\,\sin\theta_{14}}$, where $\theta_{ij}$ is the angle between lines $i$ and $j$. On the left it is 3.012836. On the right, from entirely different angles, it is 3.012836, and the two agree to eight parts in a trillion.

The cross-ratio of four lines through a point has a second reading that makes it concrete. Cross the four lines with any other line, and the four crossing points on it have a cross-ratio of their own — the one what a projection destroys found to be the quantity a projection keeps. That point cross-ratio does not depend on which line crosses them. Three transversals across the left pencil, two running across the frame at different heights and one running down it, cut the four lines at very different spacings, and all three give 3.012836387.

So the number is a property of the four lines, readable from angles or from any line across them, and the same number is a property of their four partners in the other picture. That is what “the pencils are related by a projective map” means, measured.

Why it has to be so #

The reason is the book, and it fits in a paragraph.

A page of the book is a plane through the spine. In the first picture, each page appears as the line where it cuts the picture plane, and the cut passes through the epipole because the spine pierces the picture plane there. The family of pages is itself one-dimensional — a plane through a fixed line is determined by one angle — and the map from a page to its line in the first picture is a central projection of the family of planes onto the family of lines through a point. It keeps cross-ratio, because every central projection between one-dimensional families does. The same holds for the second picture. The map from the first pencil to the second is one of those projections undone and the other applied, and a composition of maps that each keep cross-ratio keeps it too.

A line is a space of its own set out the arithmetic of exactly this kind of map on a line of points: a point of a line is one ratio, a map of the line to another is three numbers, three pairs fix it, and the cross-ratio is the invariant. A pencil of lines through a point is a line in that sense — its members are one-parameter, and projective maps between pencils are three-number maps. Nothing new has to be proved about lines of lines; they inherit everything.

A third route, through the pages themselves #

The two cross-ratios so far were read off two pictures. The book’s pages can be measured without either.

Each page is the plane through both eyes and one of the four marked points of the courtyard. Measured by how far each is turned about the line between the eyes, the four planes stand at 0°, −46.77°, −31.18° and −17.96° from the first. Their cross-ratio, computed from the sines of those turns in exactly the way the lines’ was, is 3.012836 — the same number the two pencils carry, from a computation that used the cameras’ positions and the courtyard’s points and never formed a picture.

That is the argument of the previous section checked at its root. Each picture’s pencil is a projection of the pages, and a projection keeps the cross-ratio of four pages as the cross-ratio of four lines. The two pictures agree with each other because both agree with the pages.

Why one picture shows the true angles #

Setting the three sets of angles side by side explains the half-degree agreement of the first arrangement, and it is the more interesting result.

The planes’ turns give neighbouring gaps of 46.8°, 15.6° and 13.2°. The left picture’s lines have gaps of 46.4°, 15.3° and 13.1° — within half a degree of the planes’. The right picture’s have 35.0°, 11.7° and 9.7°, which are nothing like them.

The difference between the two pictures is where the epipole sits. The left picture’s epipole is at pixel (447, 96), not far from the middle of the frame, so the left camera is looking nearly along the line between the eyes. A camera looking straight down the spine of a book sees its pages edge-on as lines radiating from the centre, each at its true angle to the others, because the picture plane is then perpendicular to every page and cuts each one at its dihedral angle. The left picture is close to that view, so it shows the pages nearly as they are.

The right picture’s epipole is at (948, 47), off the right-hand edge. The right camera is looking at the pages from well off their spine, and a plane seen obliquely foreshortens every angle in it — here by about a quarter.

So the first arrangement agreed to half a degree because both of its cameras looked nearly along the line between them, and both therefore drew the pages close to their true angles. The agreement was not the map keeping angles. It was two pictures each drawing the same true angles, and the map between them happening to be close to the identity. Turn one camera away from the spine and the angles part; the cross-ratio does not move.

Three pairs fix the map #

A map with three numbers is fixed by three pairs, which makes the fourth a prediction.

Each line in a pencil can be labelled by the tangent of its angle, and in that label a projective map between pencils has the form $t' = \frac{at + b}{ct + 1}$ — three numbers. Fitted to lines 1, 2 and 3 alone, the map predicts the angle of line 4 in the right picture, and the mark line 4 came from lies on the predicted line to $3.1 \times 10^{-10}$ px.

Nothing in that prediction knew about cameras. It used three pairs of lines and the two epipoles they pass through. The fundamental matrix was used to draw the right-hand lines in the first place, but the prediction did not use it; it would work identically with three pairs of lines traced by hand through three pairs of matched marks.

That makes a practical point about how much two views agree on. Given the two epipoles and three matched pairs of marks not on a common page, the whole pairing of epipolar lines is fixed: every other mark’s epipolar line follows. That is the fundamental matrix’s seven degrees of freedom counted a second way — two for each epipole, three for the map between the pencils — and it is why seven marks, three answers can get by with seven.

A wrong matrix keeps it too #

The last reading is the one that stops the cross-ratio from being over-read.

Rounding every mark to a whole pixel and fitting the fundamental matrix to the rounded marks, as eight points and the basis they are read in does, gives a matrix that is wrong in a measurable way. Its right-hand epipole is 13.0 px from the true one, and its epipolar lines for the four marks run up to 1.8 px from the true lines at the middle of the frame.

Its two pencils agree with each other exactly. The cross-ratio of its four lines through its own left epipole is 3.021278, and of their epipolar lines through its own right epipole 3.021278. The number differs from the true pencils’ 3.012836, because the fitted matrix’s epipoles and lines are in slightly different places, but the fitted matrix keeps it between its own two pencils as perfectly as the true matrix keeps the true one.

That is not a coincidence of this fit. Every matrix of rank two has a null vector on each side — two epipoles — and maps lines through one to lines through the other by a projective map, because that is what a rank-two three-by-three matrix does to a pencil. The pencil agreement is a property of being a fundamental matrix at all, not of being the right one.

So checking that four epipolar lines keep their cross-ratio across the two pictures verifies the rank of the matrix and nothing about whether it describes the cameras. The check that does is the one a point is a line over there makes: whether matches that took no part in the fit fall on the lines the matrix predicts.

Where this sits among the things two pictures share #

It is worth setting the pencil map beside the other structures two views share, because they fit together.

The epipoles are two points, one in each picture, and two marks off a known plane find the other eye found one of them from a plane and two raised marks. The pencil map is a three-number map between the lines through them. A plane’s homography is an eight-number map between whole pictures, and every plane’s homography — the ground’s, a wall’s, the plane at infinity’s — induces the same pencil map, because every plane cuts every page in a line and that line is imaged into both pictures on corresponding epipolar lines. That is why $F = [\mathbf{e}']_\times H$ gives the same fundamental matrix whichever plane $H$ belongs to: the planes differ in how they map points along each line, and agree on how they pair the lines.

And an epipole in the picture leaves a blind disc is what happens at the one member of each pencil that is not really a line of the pencil at all: the epipole itself, where every page meets and where a mark carries no information about which page it is on.

When the epipole goes to infinity #

The pencils in these figures converge on finite points, and the arithmetic is written with angles about those points. Nothing in the result depends on the points being finite.

When both cameras face the same way, side by side, the line between the eyes is parallel to both pictures, the epipoles go to infinity, and every pencil becomes a family of parallel lines. Four parallel lines have no angles between them, but they have a cross-ratio all the same: cut them with any line across and read the cross-ratio of the four crossings, exactly as the transversal figure does, and every transversal gives the same number. The map between the two pencils is then a map between two families of parallel lines — a one-dimensional projectivity on their heights — and it still has three numbers and still keeps that cross-ratio.

For the special case in which the two pictures are also aligned, the map is the identity: the epipolar line of a mark at some row is the same row in the other picture. That is the arrangement depth is a reciprocal works in, and it is why a matcher there searches one row. The pencil map has not disappeared in that case; it has been made trivial, and rectification is a family, not an operation measures what it costs to make it so for a pair that was not built that way.

What this does not settle #

Four marks. The cross-ratio is computed for one choice of four marks, spread round the left epipole to keep the lines well apart. Four lines crowded into a few degrees have a cross-ratio that is exact in arithmetic and very sensitive to reading, and that sensitivity was not swept.

Exact epipoles for the exact case. The first four figures use the cameras’ own fundamental matrix, so the epipoles and lines are exact. Only the last figure fits a matrix to rounded marks. How precisely a cross-ratio of four epipolar lines can be read from marks alone, without a fitted matrix, is not measured.

A pair chosen for asymmetry. The pair was built to make the angles change. On a symmetric pair they change by a tenth of a degree, and that is recorded above as the reason for the choice.

Still open: which pairs of lines rectify #

The pencil map is the part of the fundamental matrix that the rest of stereo is built on without saying so. Rectifying a pair — warping both pictures so that epipolar lines become horizontal rows at matching heights — is a choice of two homographies, one per picture, that send both epipoles to infinity along the horizontal and make the pencil map the identity on rows.

The question that leaves is how many ways there are to do that. Sending an epipole to infinity along a row fixes some of each homography’s eight numbers and leaves others free, and every choice of the free ones gives a rectified pair with the same pencil map and therefore, it would seem, the same depth for every match. Whether the depths really are identical across the family, and how much the free choices change what happens to the pixels — how far each warp stretches or squeezes the picture it resamples — is the measurement that says whether rectification is one operation or a family of them that differ in what they cost the picture.