One depth per sample is not enough

The precision a depth buffer has left measured what a depth buffer’s codes are spent on, and found the answer to be a reciprocal: most of them go to the first few per cent of the view. This essay is about a different limit of the same buffer, and it is not a precision limit at all.

A depth buffer holds one number per sample. At most surfaces that is the whole truth. At an occluding edge it is not, and the shortfall is not small.

What a post-process blur does #

Every real-time renderer computes depth of field the same way. Render the scene sharp, keep the depth buffer, and for each pixel gather from its neighbours with a radius taken from that pixel’s own depth.

The radius is right: the centre has an area gives it as R|1 − v₀/v(Z)|, and a renderer that computes it from the buffer computes the correct number — which is also the number the sharp band is a decision cuts its band out of. The gather is what is wrong, and it is wrong for a reason that has nothing to do with how many samples it takes.

The measurement #

An 85 mm lens at f/1.4, an opaque edge at 1.2 metres, a background at the focus distance of six.

The pupil’s own answer is computed by integrating over the disc with the occlusion applied ray by ray. The renderer’s answer is computed the way a renderer computes it: at each sample, take the depth of whatever surface is there — the near one on one side of the edge, the far one on the other — and gather with that radius. The near surface’s radius is 33.5 pixels; the far one’s is 0.0, because the background is exactly in focus.

The two profiles part company by 70 per cent of full scale, over a band 10.5 pixels wide.

And the sign of the difference is always the same. The gather cannot let the background through, because the sharp image it is gathering from does not contain the background behind the edge.

The control #

A profile computed at sixty-one sample positions across a sharp step has one sample straddling it, where the pupil integral reads a half and a gather reads nothing. That is a single-sample artefact, not a disagreement about the picture, and a control stated on the peak difference would have been reporting it.

So the control is stated on the width instead. With the edge in focus and the background just behind it, the band over which the two disagree is 1.4 pixels — the straddling samples — against 10.5 pixels in the defocused case. A factor of seven, and the two disagreements are different in kind rather than in degree.

Getting that wrong once is what the control is for. The first version of this measurement compared peak differences and reported the focused case at fifty per cent, which is a picture of the sampling grid.

What the buffer would need to hold #

The honest answer is not “a second depth”. It is a list, and the list is unbounded.

At a pixel near a defocused edge, the light arriving comes from every surface visible from any point of the pupil. Two surfaces is the simple case; a defocused chain-link fence in front of a defocused hedge in front of a sharp building is three, and there is no bound. Storing per-sample lists is what a deep framebuffer or an A-buffer does, and it is why those exist.

There is a second thing the buffer would need and it is easier to miss: visibility from each pupil position, not merely from the pupil’s centre. A deep buffer built by rasterising from the centre records the surfaces the centre can see, and the whole content of a pupil sees around an edge is that the pupil sees surfaces the centre cannot. So even a deep buffer from one viewpoint is short of the answer.

That is what makes this an information problem rather than a storage problem, and it is why the only exact method is to sample the pupil.

The deficit is an amount of light, and it is conserved #

The disagreement is quoted as a peak and a width, and there is a third number that is more useful than either because no tuning of the gather can change it.

Across the transition the pupil’s answer at each position is the background’s brightness times the open share — the fraction of the pupil that sees past the edge — which runs from 0 on the occluded side to 1 on the clear side. Integrating that share across the strip gives exactly half the strip’s width, because the share’s profile is antisymmetric about its midpoint. So the light the pupil delivers that the gather cannot is

$$\tfrac{1}{2} \times (\text{strip width}) \times (\text{background brightness}),$$

a fixed quantity, set by the geometry and the scene and by nothing about the blur.

That is a stronger statement than the peak or the width, and the reason is that it survives every mitigation. A depth-aware weighting redistributes light within the frame; a foreground extension copies light from elsewhere; a scatter formulation changes which pixels spread where. None of them changes the total, because none of them has access to the hidden strip’s actual contents, and a redistribution conserves what it redistributes. So a gather-based renderer’s defocused edges are short by a computable amount of energy, always, whatever is done to the kernel.

It also supplies a test that is cheaper than comparing profiles. Render a scene twice — once with a per-pixel gather and once with a sampled pupil — and integrate the total light over a window containing a defocused edge. The two profiles may be argued about; the two totals differ by a number the geometry predicts in advance, and a mitigation that closes the gap has necessarily taken the light from somewhere else in the frame, which the same integral over a wider window will show.

That is the form this collection prefers for any impossibility claim, and it is the form the two-background demonstration already takes one level up. A profile comparison invites a better kernel. A conserved quantity that the input does not contain rules out every kernel at once, and it does so with an integral rather than an argument.

Why it looks acceptable anyway #

A fair question, since every game ships this and few players complain.

Most edges are between similar depths. The failure’s size scales with R(Z₂/Z₁ − 1), so it is large only when a strongly defocused near object stands in front of a much more distant one — which in a typical scene is a minority of edges.

The error is in the direction of “too opaque”. A gather makes a defocused foreground more solid than it should be, and a viewer with no reference has no way to know. The failure is not a visible artefact; it is a systematic absence of one.

And the artefacts people do complain about are different ones. Bleeding of a sharp foreground onto a blurred background, and the halo where the gather radius jumps across an edge, are both consequences of the same missing information but present as visible defects rather than as absences, so they attract fixes — depth-aware weighting, layer separation, foreground extension — that ameliorate the symptom without addressing the cause.

That last point is the useful one. Every mitigation in the standard toolbox is an attempt to guess the hidden background, and none of them can be right, because the input does not determine the answer.

The same shape, three times in this field #

This is the third time this collection has found a rendering approximation failing at a discontinuity, and putting them together says what the pattern is.

Perspective-correct interpolation. A texture does not interpolate on the page — a quantity that is linear in the world is not linear in the picture, and interpolating it on the page is wrong by an amount that grows with the depth range across the primitive.

Depth precision. The precision a depth buffer has left — the codes are spent reciprocally, so a distant surface has fewer of them than a near one and coplanar surfaces fight.

And this. One value per sample cannot represent what a finite aperture integrates over.

All three are the same trade: the pipeline stores a rasterised, single-valued, page-space approximation of something that is continuous, multi-valued and world-space, and each of them fails exactly where the approximation’s assumption fails — at large depth ranges, at discontinuities, at edges.

What is actually done about it #

Three approaches, in ascending order of cost and correctness.

Layered blur. Split the frame into a few depth layers, blur each independently, and composite back to front with the foreground layers’ edges extended inward. The extension is a guess at the hidden background — usually the nearest visible background pixel, stretched — and it is right when the background is smooth and wrong when it is not.

Gathering with scatter semantics. Let each pixel spread into its neighbours rather than gathering from them, which gets the foreground’s transparency approximately right because a defocused foreground pixel really does spread. This handles the occlusion direction correctly and still cannot produce hidden background.

And sampling the pupil. Render the scene several times from different points across the pupil and average, which is exact in the limit and is what offline renderers do. Sixteen samples removes most of the visible error; the number needed for a smooth result grows with the disc’s area.

The third is the only one that answers the question rather than approximating an answer, and it is expensive for exactly the reason this essay is about: it is gathering information that one pass does not have.

What it costs a measurement rather than a picture #

Rendering is where this matters most often; measurement is where it matters most.

A synthetic image used as ground truth for a vision algorithm inherits whatever its renderer got wrong. If the renderer’s defocus is a per-pixel gather, every defocused edge in the training set has the wrong profile — too opaque, in a consistent direction — and a method trained on it learns that bias.

The same applies to any evaluation of a depth-from-defocus method against rendered data: the method is being scored against images whose edges do not have the profile a camera would produce, and its errors at edges are being measured against the wrong answer. That is not hypothetical; edges are exactly where such methods are weakest and where the evaluation is therefore least trustworthy.

The remedy is the same as the rendering remedy — sample the pupil — and it is much cheaper here, because ground truth is generated offline and sixteen or sixty-four samples costs time rather than frame rate.

Why this belongs to the pipeline field and not the sensor field #

A boundary worth drawing, since the geometry is the aperture row’s and the failure is a renderer’s.

The aperture row measures what an instrument does. This measures what a representation cannot do, and the representation is a depth buffer — a rasterised grid of one depth per sample, which is a decision made in hardware in the nineteen-seventies and inherited by everything since.

That makes it the same kind of finding as this field’s others: not “the arithmetic is wrong” but “the data structure cannot hold the answer”. What happens behind the eye is the same shape — a clipper exists because homogeneous coordinates cannot represent a point behind the camera without one — and so is a pixel is not a point.

Each of them is a place where the pipeline’s own representation, not its precision, sets the limit.

The two radii, and why one of them is zero here #

Worth pausing on the arrangement’s numbers, because one of them looks like a special case and is the ordinary one.

The near surface’s gather radius is 33.5 pixels and the far surface’s is 0.0 — because the background sits exactly at the focus distance. That is not a contrivance; it is the commonest arrangement in a photograph, where the subject is in focus and the foreground obstruction is not.

It also makes the failure as clean as possible to state. Every pixel on the background side of the edge is gathered with a radius of zero, so the renderer’s profile there is the sharp image itself: a step. The pupil’s profile is not a step, because part of the pupil sees background that the step’s own occluder is covering. The whole disagreement is therefore on the background side and it is the full height of the transition.

Move the background off the focus distance and the far radius becomes nonzero, the renderer’s profile softens, and the disagreement shrinks without disappearing — because a soft wrong answer is still wrong. The band narrows and the peak falls; the sign does not change.

What a hardware designer would have to change #

Not a proposal, but worth naming, because the shape of the fix says what the limit is.

The buffer is a grid of samples with one depth each because rasterisation visits triangles and writes a fixed-size record per covered sample. Anything richer — a list per sample, a set of fragments, a per-sample visibility function — breaks the fixed-size record, which is what makes the whole pipeline’s memory traffic predictable.

Deep framebuffers exist and are used offline for exactly this class of problem, and they are not adopted in real time for the memory reason rather than a computational one. What they still would not give is visibility from anywhere but the centre of the pupil, so the honest fix is not a richer buffer but more viewpoints, which is the same conclusion the software side reaches.

That is a satisfying place to end, because it says the limit is not in the buffer’s width. It is in the number of places the scene was looked at from, and one is not enough.

What the sixty-one samples are #

The profiles are computed at sixty-one positions across a span of about a millimetre on the sensor, which is roughly twenty pixels of the picture, and that resolution is chosen rather than default.

It has to be fine enough that the pupil’s own profile — which reaches from zero to one over the disc’s diameter — is resolved by many samples, and coarse enough that the pupil integral, which is a ninety-six by ninety-six sweep of the disc at every position, finishes in a reasonable time. Sixty-one across a thirty-three-pixel disc gives about two samples per pixel of blur, which resolves the profile and leaves the straddling artefact at the sharp edge visible rather than hidden.

Reporting that artefact rather than smoothing it away is what let the control be stated on a width, and it is the reason the number quoted for the in-focus case is 1.4 pixels rather than zero.

The short version #

A depth buffer holds one depth per sample. A post-process blur can therefore ask only how far away the surface at each pixel is, and at an occluding edge the correct answer involves two surfaces and what the pupil can see past one of them.

Measured on an 85 mm at f/1.4 with an edge at 1.2 metres and a background at six: the gather and the pupil’s own integral differ by 70 per cent of full scale over a band 10.5 pixels wide, against a 1.4-pixel control with the edge in focus. The difference always has the same sign, because the sharp image the gather reads has no pixel for the background behind the edge.

No amount of storage fixes it from one viewpoint. Sampling the pupil does, and that is exactly the information the extra samples are for.