The track and the scene together

The site’s oldest result is a round trip. A box is drawn from a stated camera; the camera is thrown away; and the focal length is recovered from the twelve drawn edges alone, agreeing with the original to one part in 10¹⁵.

This is that trip at its largest. Six photographs of a courtyard, forty-four points, and the recovery is asked for everything at once: where each camera stood, how each one was turned, and where every point in the scene is.

What is being solved for #

The parameter count is worth spelling out, because it is unusually large for a figure on this site and because its structure matters later.

Each camera has six numbers: three for its orientation, written as an axis-angle vector, and three for its position. Six cameras give thirty-six. Each scene point has three. Forty-four points give a hundred and thirty-two. The total is a hundred and sixty-eight.

Against that, the observations. Every camera that can see a point contributes one observation, and each observation is two numbers — a column and a row in that camera’s picture. There are one hundred and ninety-two observations here, so three hundred and eighty-four measured numbers constraining a hundred and sixty-eight unknowns.

The problem is therefore overdetermined by a comfortable margin, which is what makes it solvable, and the margin is where the accuracy comes from: each point is seen several times, and its position has to be consistent with all of those views at once.

The quantity minimised is the reprojection error — for each observation, the distance in pixels between where the mark was seen and where the current estimate of the camera and the point would put it. That is the right quantity for the reason the triangulation essay gave: the errors occurred in pixels, so a good explanation of the data is one that is off by few pixels, not one that is near some rays in metres.

Why the observations are not all of them #

One number in the count above hides a decision. There are six cameras and forty-four points, which would be two hundred and sixty-four observations if every camera saw every point; there are one hundred and ninety-two.

The missing seventy-two are points that fall outside a camera’s frame or behind it. The solver is told about the observations that exist and nothing about the ones that do not, which is the correct treatment and is not the only one available — a formulation that penalised a point for not being seen would be inventing evidence out of the frame’s edge.

The consequence is that the constraint on a point is uneven. A point seen by all six cameras is very well determined; one seen by two, at the edge of the sweep, is determined about as well as a two-view triangulation manages. The reported reprojection error averages over all of them and says nothing about the distribution, so a scene with a few weakly-seen points and many strongly-seen ones reports a small number and contains a few positions that are barely constrained at all.

This is the structural reason a reconstruction is usually reported with a per-point count as well as an error, and it is worth knowing that the pretty aggregate figure hides it. It is also why the figure’s scene was built to be seen from everywhere rather than to be realistic: a courtyard of four blocks and two rows of ground marks, viewed along a shallow arc, keeps most points visible in most views, which makes the measurement about the geometry rather than about which points happened to be occluded.

Rotations as three numbers #

One representation choice deserves a paragraph, because getting it wrong would bury the next essay’s entire result.

A rotation could be carried as a 3×3 matrix, which is nine numbers. It is not, because a rotation has three degrees of freedom and the other six are the constraints that keep the matrix orthogonal. A solver parameterised by matrix entries would have six spurious directions per camera — the ones that leave the space of rotations — and would wander off into matrices that are not rotations unless prevented.

More to the point for this field: those spurious directions would appear in the Jacobian’s null space alongside the ones that are actually free, and with six cameras that is thirty-six extra flat directions burying the seven that are the field’s whole result.

So rotations are carried as axis-angle vectors — three numbers, converted to a matrix by Rodrigues’ formula when needed. The parameterisation has its own awkwardness at a half-turn, where the antisymmetric part of the matrix carries no information about the axis and the inverse conversion has to take it from the symmetric part instead. That case is rare and is handled explicitly, because a half-turn is exactly what a rejected pose looks like and a silently wrong conversion there would be hard to trace.

The Jacobian, by differences #

The solver needs the derivative of every residual with respect to every parameter — a 384 × 168 matrix, recomputed at each iteration.

It is computed by central differences rather than analytically, and the choice is deliberate. The analytic derivative of a projection through a Rodrigues rotation is several pages of chain rule, every line of which is an opportunity to write a Jacobian that is nearly right.

A nearly-right Jacobian is a peculiarly bad thing to have. It does not make the solver fail: Gauss–Newton with a slightly wrong derivative still converges, just more slowly, and the answer it converges to is still the minimum of the true objective because the objective is evaluated exactly. So the symptom of a subtly wrong analytic Jacobian is that the solve takes eleven iterations instead of five, which nobody investigates.

But this field’s central measurement is the Jacobian’s null space, and a nearly-right Jacobian has a nearly-right null space. The number that this field exists to report would be quietly wrong, with no other symptom at all. A central difference is slower and cannot be subtly wrong, and the trade is obvious once the measurement is the point rather than the speed.

The step size is scaled to each parameter, because rotations are in radians and translations in metres and a single absolute step cannot suit both.

Damping, and why it is not a detail #

The normal equations of this problem are singular. Exactly, and by seven.

That is the next essay’s subject and it has an immediate practical consequence here: an undamped Gauss–Newton step asks a singular system for an answer, and gets one only by accident.

Levenberg–Marquardt’s damping term is what makes the solve possible. Adding a multiple of the identity to the normal matrix moves the seven flat directions off zero, so the system becomes solvable, and the step simply does not travel along them. That is the numerically honest form of “pick a gauge”: rather than fixing seven parameters by decree, the solver declines to move in the directions that do not change anything.

The damping is adapted as the solve proceeds — reduced when a step improves the error, increased when it does not — which is standard and is what lets the run in the figure converge in five iterations from a starting point that was several pixels wrong.

The dense solve used here is not what a large problem would use. Real bundle adjustment exploits the sparsity of the Jacobian — each observation touches one camera and one point, so the normal matrix has a characteristic block structure, and eliminating the point block first (the Schur complement) reduces the work enormously. That machinery is numerical-linear-algebra.com’s subject and is licensed rather than taken; at this figure’s size the dense solve is exact and says exactly the same thing about the geometry.

Where the solve starts #

A non-linear solve needs a starting point, and where it comes from is not a detail: an optimiser finds a nearby minimum, and a badly chosen start finds a different one or none at all.

The start used here is built by the previous field. Take the views in sequence, recover the relative pose of each consecutive pair from its correspondences, compose them along the chain to get an approximate pose for every camera, and triangulate every point in that frame. The result is a complete reconstruction — every parameter has a value — and it is wrong by several pixels of reprojection.

That is exactly what a starting point should be. It is in the right basin, so the solver descends to the right minimum; it is visibly wrong, so the improvement is a real improvement rather than a confirmation.

There is one piece of scaffolding in it worth naming, because it would otherwise be invisible cheating. A two-view pair supplies the direction of its baseline and not the length, so a chain of pairwise poses has an unknown scale per link, and composing them requires those scales to be consistent. The figure supplies each link’s true baseline length for that purpose. That is a genuine input the pictures do not contain, and it is used only to build the starting point — the adjustment that follows is free to move every camera and every point, and the accuracy it reaches is measured against a track whose scale has been removed anyway.

The proper way to resolve those scales from the pictures alone is to triangulate points shared between consecutive pairs and require them to agree, which is how a real sequential pipeline works. It is more machinery than this figure needs to make its point, and it is recorded here rather than done, so that the one supplied quantity is on the record rather than buried in a helper.

The comparison that had to remove something first #

The recovered camera centres agree with the true ones to 2.0 × 10⁻³ of the track’s own mean radius. The phrasing of that number is the interesting part.

A direct comparison — recovered position minus true position, in metres — would report a large error on a perfect reconstruction. The reconstruction is written in a frame of its own choosing and at a scale of its own choosing, so its coordinates are a statement about the convention rather than about the geometry.

So the comparison is done after removing the freedom. Both tracks are centred, both are scaled to unit mean radius, and the rotation between them is found by the orthogonal Procrustes solution — the singular value decomposition that gives the best rotation aligning two point sets. What is left after that is genuine disagreement, and it is 2 × 10⁻³.

The units are therefore “fractions of the track’s mean radius” rather than metres, which is not evasion. It is the only scale-free way to state the result, and a number in metres would be a number about the arbitrary scale factor as much as about the recovery.

That removal is not a special step for this figure. It is the operation the whole next essay is about — the seven-parameter family that a reconstruction is determined only up to — and doing it before any comparison is the standing discipline of the field.

Everything moving at once #

The word doing the most work in this essay’s title is together, and it is worth being explicit about what the alternative would be.

The obvious way to solve a problem of this shape is in stages. Fix the cameras and solve for the points, which is a triangulation per point and is independent across points. Then fix the points and solve for the cameras, which is a pose problem per camera and is independent across cameras. Alternate.

That converges, it is easy to implement, and it converges slowly — because the shape of the error surface near the minimum couples cameras to points, and an algorithm that only ever moves along the coordinate axes of that coupling has to zigzag. A small error in a camera’s position is very nearly compensated by a correlated small error in the positions of every point it sees, and the pair of errors together produces almost no reprojection error at all. Moving only one of them at a time climbs out of a valley by walking along its walls.

Bundle adjustment moves all one hundred and sixty-eight parameters in one step, along the direction the coupled system actually wants. That is why five iterations suffice, and it is the reason the method is named for the bundle — the whole assembly of rays from every camera to every point, adjusted as one object.

It is also why the seven flat directions are a fact about the joint problem rather than about any camera or any point. Each of them moves every parameter at once, in a coordinated way, and none of them is visible from inside a single camera’s pose problem or a single point’s triangulation. An alternating solver would never encounter them as a group; it would merely converge oddly and never say why.

Why a plan rather than a perspective #

The figure draws the courtyard and the camera track from above.

A perspective view would be more attractive and would be dishonest in a specific way: it would be a picture of a reconstruction, drawn from a viewpoint, and the viewpoint would be a choice made by the figure rather than by the data. Worse, foreshortening would make the agreement between the recovered and true tracks look better or worse depending on where the viewpoint was put.

A plan shows the two tracks at the same scale in the plane they mostly live in, and the disagreement between them is visible at true size. It is the same reasoning the shape-and-size figure used, and it comes up on this site often enough to be a habit: when the measurement is a distance, draw it in a projection that preserves distances.

What has been established #

Six photographs, read to a pixel, determine the positions and orientations of the six cameras that took them and the positions of forty-four points, to two parts in a thousand of the arrangement’s own size, with no external information whatever.

That is a strong result and it is worth stating carefully, because the qualifiers are not decoration.

Read to a pixel is what sets the two parts in a thousand. Exact marks give machine precision; coarser marks give worse.

Of the arrangement’s own size is the scale ambiguity, unchanged from two views. Six pictures do not fix a size any more than two do.

With no external information is the claim that makes it a round trip rather than a fit. The solver was handed 384 numbers, all of them positions in pictures, and it produced a camera track and a scene.

And to two parts in a thousand is a comparison made after removing seven degrees of freedom that the pictures do not determine. Which raises the obvious question of why there are seven, and whether they are genuinely free or merely poorly determined — a question that turns out to have a very sharp answer.

There is one more thing worth saying about what has just happened, because it is easy to lose among the parameter counts. The first essay on this site recovered one camera from one drawing. This one recovers six cameras and a scene from six drawings, using no new principle — the same projection, the same insistence that the recovery is never shown what it is trying to find, and the same demand that the answer be compared against something it was not given.

What changed is only the size of the system, and the interesting consequence of that change is not that it got harder. It is that the answer acquired a qualifier it did not have at size one. A single camera recovered from a single picture is recovered outright, to fifteen digits, with nothing left over. Six cameras recovered from six pictures are recovered up to a similarity, and no amount of care removes it. Growing the problem did not merely make it more expensive; it made the answer a different kind of object.