Where the adjustment stops

A converged optimiser produces a number, and the number is usually reported as though it were a quality score. Lower is better; zero would be perfect.

For a bundle adjustment on real measurements, zero would be a symptom.

Two curves, two floors #

The two runs differ in one respect. One is given correspondences at full double precision; the other is given the same correspondences rounded to a whole pixel. Everything else — the cameras, the scene, the starting point, the solver, the damping schedule — is identical.

The exact run falls to 1.3 × 10⁻¹¹ px. That is not a measurement of anything geometric. It is where double-precision arithmetic runs out on a problem of this size, and a run that stopped an order of magnitude higher would be reporting the solver’s stopping rule rather than the data.

The quantised run falls to 0.3324 px and stops. It stops because there is nowhere further to go: no assignment of six camera poses and forty-four point positions reprojects a set of quantised marks exactly, and the residual left over is the part of the rounding that no consistent geometry can explain.

Why the floor is not zero #

The reason is worth stating carefully, because “noise in the data” is the right answer given imprecisely.

Rounding each mark to a whole pixel moves it by up to half a pixel in each coordinate. Those displacements are not consistent with any camera geometry — they are independent per observation, and a projection is a highly structured map. So the rounded observations are not the projections of any scene from any cameras.

The best the solver can do is find the geometry whose projections come closest, in the least-squares sense, and what is left is the component of the rounding that lies outside the space of achievable projections.

There is a rough accounting available. With 384 residuals and 168 parameters, of which 7 are free, 161 are genuinely fitted. So about 223 residual degrees of freedom remain, and the rounding error can be absorbed only in the 161 directions the parameters can reach. A uniform rounding to a whole pixel has a standard deviation of about 0.29 px per coordinate, and a fit that removes roughly 40% of the directions leaves a residual on the order of a quarter of a pixel. Measured: 0.3324.

That is agreement at the level of an order-of-magnitude sanity check rather than a derivation, and it is quoted as such. What it establishes is that the floor is where it should be — the residual is the unexplainable part of the input noise, not a failure to converge.

Why zero would be a symptom #

Suppose the same run had reached 10⁻¹¹ on the quantised marks.

That would mean the solver had found a geometry which projects exactly onto a set of positions that were produced by rounding. Since no genuine geometry does that, the geometry it found would have to be absorbing the rounding into its parameters — which is to say, it would have moved the cameras and the points to accommodate errors that are not features of the scene.

That is overfitting, and its symptom is the opposite of what an inspector would look for: a better residual, from a worse reconstruction.

The way it would happen in practice is not mysterious. Free enough parameters — let the focal lengths vary per view, let the principal points move, add distortion terms — and a system with 384 residuals can be given enough freedom to fit almost anything. Each added parameter reduces the residual, none of them announces that it has stopped describing the instrument, and the reconstruction quietly gets worse while the reported number improves.

This site has measured that exact failure once already, on a different problem. Fitting one distortion coefficient to data from a two-term lens returned a residual of 0.68 px and a coefficient 22% wrong, and freeing the second coefficient made the residual smaller while the two coefficients became correlated at −0.997 and individually meaningless. Small residual, wrong parameters, and no diagnostic in the residual itself.

The residual and the reconstruction move differently #

There is a check available in this figure that a single run cannot provide, and it makes the point sharper than the accounting does.

Compare the two runs on the quantity that actually matters — how close the recovered camera track is to the true one — rather than on the residual. The exact run recovers it to machine precision. The quantised run recovers it to about two parts in a thousand of the track’s own size.

Now compare the residuals: 1.3 × 10⁻¹¹ px against 0.33 px, a ratio of about 2.5 × 10¹⁰.

The two ratios are wildly different, and they have to be. The residual measures how well the model explains the marks it was given; the track error measures how close the model is to the truth. Rounding the marks degraded the first by ten orders of magnitude and the second by three, because most of the rounding error is absorbable into a reconstruction that is still nearly right.

That is the useful form of the warning. The residual is not a monotone proxy for accuracy, and the two are not even the same order of magnitude apart. A reader given only the residual has no way to convert it into a statement about the reconstruction, and a reader given only the track error would not know whether the model fitted.

It also explains why the quantised run’s residual, though ten orders of magnitude larger, is not alarming. A third of a pixel is what a whole-pixel reading should leave. The reconstruction underneath it is good to two parts in a thousand, and nothing about the residual would have said so.

What to check instead #

If the residual is not a quality score, three things are.

Whether it sits where the noise says it should. That is the accounting above. A residual far below the input noise is overfitting; far above it is a model that does not fit — an outlier, an uncorrected distortion, a wrong calibration.

Whether the parameters are determined. The Jacobian’s spectrum answers this, and it answers more than the gauge question: any direction with a small singular value beyond the seven is a combination of parameters the data barely constrains, and its estimate is whatever the starting point suggested. Seven is expected; an eighth would be a warning.

And whether the residual is distributed as noise should be. One large residual among many small ones is an outlier, not a small increase in the aggregate. A residual pattern that is spatially coherent — larger toward the edges of every frame, say — is a systematic effect the model is missing, and radial distortion is the usual culprit.

None of the three is the aggregate number, and all three are computable from what a solver already has.

What the convergence curve shows #

The shape of the two curves is worth reading as well as their endpoints.

Both fall steeply for a few iterations and then flatten. That is characteristic of a damped Gauss–Newton method near a good starting point: the first steps are large and reduce the error by orders of magnitude, and once the quadratic model is accurate the method converges fast, so the interesting part of the run is over quickly.

Five to six iterations is what both runs take. That is a consequence of the starting point being in the right basin — the chained pairwise reconstruction is wrong by a few pixels and right in structure — and it is worth contrasting with what happens without one. Started from a random guess, the same solver does not converge at all: the reprojection error surface for a hundred and sixty-eight parameters has a great many local minima, and almost all of them correspond to configurations that are not reconstructions of anything.

So the curve is short, and its shortness is a fact about the initialisation rather than about the method’s power.

What the exact run is for #

It would be reasonable to ask why the exact curve is drawn at all, given that no real measurement is exact and the whole essay argues that the interesting floor is the other one.

It is drawn because it is the only way to separate two things that a single curve conflates.

A converged run stops at some value. That value is the sum of two contributions: the part of the input noise that no geometry can explain, and whatever error the implementation itself carries — a wrong Jacobian, an accumulating rounding, a convergence criterion that stops early, a bug in the projection.

The exact run isolates the second. With no input noise at all, everything remaining is the implementation, and it comes to 1.3 × 10⁻¹¹ px. That is small enough to be neglected beside a third of a pixel, and knowing it is small is what licenses attributing the quantised run’s floor entirely to the data.

Without that check, the 0.33 px would be a number with two possible explanations and no way to choose between them. A solver with a subtly wrong derivative, or a projection with a sign error somewhere harmless, would also stop somewhere plausible — and the plausibility is the problem, because a third of a pixel is exactly the sort of number that looks like noise.

This is the same discipline as the site’s dry controls in the refraction field, where every wet measurement is computed beside the same measurement with the water removed, by the same code from the same scene. The control is not the result. It is what makes the result attributable.

The damping, seen in the run #

One further detail is visible in the run and worth explaining, because it is what makes convergence possible on a singular system.

Levenberg–Marquardt carries a damping parameter that is reduced when a step succeeds and increased when it fails. In the runs here it starts modest, falls quickly as the early steps succeed, and the method becomes nearly pure Gauss–Newton for the final iterations.

Its role is not merely to stabilise. The normal equations of this problem are singular by exactly seven, so an undamped step is asking a singular system for an answer. The damping moves those seven directions off zero, and the resulting step has almost no component along them — the solver declines to travel in the directions that buy nothing, without anybody having to fix a gauge by decree.

That is why the recovered reconstruction ends up wherever its starting point put it, in position, orientation and scale, and why every comparison in this field aligns before it measures.

Rounding is not the noise a real measurement has #

One honesty about the model of error used throughout this phase, since this essay is where it does the most work.

Quantising a mark to a pixel grid is a deterministic operation. The same mark always rounds the same way, there is no random number generator anywhere in these figures, and every number in them is reproducible exactly. That is a real advantage: a figure whose measurements depend on a seed is a figure whose measurements nobody else can check, and this site’s habit is that a number should be recomputable rather than merely reported.

It is also not what a real matcher produces. Rounding error is bounded, uniform, and independent between marks. A real matcher’s error is unbounded — a mismatch is arbitrarily large — usually not uniform, and correlated between nearby marks, because neighbouring patches of a picture share whatever made one of them hard to locate. It is also correlated between the two views of a stereo pair in a way that partly cancels, which is what makes sub-pixel disparity possible at all.

So the floors measured here are the floors for a particular, tidy error model. The structure of the argument does not depend on which model is used — a residual sits where the input noise says it should, and a solver that beats that is fitting the noise — but the specific value 0.3324 px is about rounding, and a real reconstruction with a real matcher would land somewhere else for reasons this phase does not measure.

Where the difference matters most is the correlation. Independent errors average down as more observations are added; correlated ones do not, and a residual that assumes independence will look better than the reconstruction deserves. That is a systematic optimism rather than a random one, and it is one more reason not to read the residual as an accuracy.

The honest way to report a fit #

Pulling it together, the number to publish alongside a reconstruction is not one number.

The residual, stated in pixels, with the measurement precision it should be compared against. The count of parameters and observations, so a reader can see how overdetermined the problem was. The gauge count and the gap, so a reader can see that nothing beyond the expected seven was free. And the distribution of per-observation residuals, or at least its worst value, so that one bad correspondence is visible rather than averaged in.

That is more work than quoting a single figure, and it is the difference between a reconstruction that has been checked and one that has converged. This site’s standing habit — every claim is given a test it could fail — has a specific form for a fitted model, and it is that the fit’s own residual is never that test. A residual measures agreement between a model and the data it was fitted to, and every failure mode in this phase produced a small one.

It is worth listing them once, because the list is the argument. A reconstruction from a two-millimetre baseline with quantised marks reprojects beautifully and is six billion times further from the truth than the exact recovery. A fit containing one bad correspondence reprojects to a fraction of a pixel and has moved the geometry for every point. A chained track is four to nine times worse than the same data supports. And an over-parameterised model reprojects better than a correct one by construction.

Four failure modes, four small residuals. The number that would have caught each of them is different in every case — a singular value, a residual distribution, a comparison against an adjustment, a parameter count against an error budget — and none of them is the number the solver prints when it stops.