What a 360-degree photograph actually is

Open the file a spherical camera writes and there is a single rectangle, twice as wide as it is tall, with the world smeared across it: the floor stretched into a band along the bottom, the ceiling into a band along the top, and everything in between compressed toward the middle. It is the least attractive image format in general use, and it is what essentially every 360° photograph and video is stored in.

The rectangle is an equirectangular projection. Its rule is as simple as a rule can be: the horizontal coordinate is the azimuth, the angle turned through; the vertical coordinate is the elevation, the angle looked up or down. Two angles, two coordinates, nothing else.

It preserves nothing, and the numbers say so exactly #

Every other surface in this family has something to point at. The flat plane keeps straight lines straight, exactly. Stereographic keeps every angle and every shape, exactly. The equal-area fisheye keeps area, exactly, to 1.0000 across the whole field. The equidistant fisheye keeps the picture’s radius proportional to angle, which is not a property of the world but is at least a property.

The equirectangular surface keeps nothing.

Its bend of a straight line is 5.7% of the line’s own chord — worse than every surface here except the cylinder. Its anisotropy is sec φ, so a shape 45° up is 41% taller than it should be and one at 60° is doubled. Its area scale is cos φ relative to the equator, which goes to zero at the poles: the top row of pixels in an equirectangular file, all of them, correspond to a single direction, straight up.

That last one is worth stating in its own sentence, because it is the format’s most conspicuous feature and its most misunderstood. The whole top edge of the rectangle is one point. A 4096-pixel-wide file spends four thousand and ninety-six pixels describing the zenith, and four thousand and ninety-six describing the nadir, and the same four thousand and ninety-six describing the horizon — where they are actually needed. Somewhere between a third and a half of the file is redundancy at the poles.

The cylinder in disguise #

There is an exact relationship between this surface and the cylindrical panorama that neither’s usual description mentions, and it changes what “these are different projections” means.

The two have identically the same anisotropy. Both stretch by sec φ, at every elevation, to fourteen digits — the two curves in the figure below lie on top of one another and the check that they do requires them to agree to 10⁻⁹.

The reason is one line. On the sphere the metric is cos²φ dθ² + dφ². Equirectangular takes (u, v) = (θ, φ), magnifying horizontally by sec φ and vertically by 1. The cylinder takes (u, v) = (θ, tan φ), magnifying horizontally by sec φ and vertically by sec²φ. The ratio of the two magnifications is sec φ in both cases, because stretching one axis by a function of that same axis leaves the ratio alone.

So the two surfaces distort shape identically. What separates them is reach and area: the cylinder’s vertical coordinate is unbounded and it can never contain the poles; the equirectangular surface’s is bounded by ±π/2 and contains everything. If a picture is going to hold the whole sphere, the cylinder is not available, and the simplest thing that is available is this.

Why the worst format is the right one #

The case for equirectangular is not a case about pictures at all, and it is completely convincing once stated.

An equirectangular file is a lookup table indexed by direction. To find the colour of a ray, take its azimuth and its elevation, scale them to pixel coordinates by two multiplications, and read. There is no trigonometry in the lookup, no square root, no branch, and the addressing is separable — one coordinate depends on azimuth only and the other on elevation only.

That matters because a 360° image is almost never displayed as it is stored. It is displayed by cutting a view out of it: a viewer picks a direction and a field of view, a renderer builds a flat picture, and for each pixel of that flat picture it needs the colour of one ray. The operation performed millions of times a second is ray in, colour out, and equirectangular is the format that makes that operation cheapest.

Compare the alternatives. Reading a ray from a stereographic file needs a half-angle tangent. From an equal-area file, an arcsine. From a cube map — the other format in real use — a branch to decide which face, then a division. Equirectangular needs two multiplications and a clamp.

The distortion the format inflicts is therefore paid once, at capture, and it is undone at every view. Nobody looks at the rectangle. The rectangle is the tape, and the picture is what comes off it.

A picture, or a record of one #

That distinction is worth pressing, because it is where this essay’s title stops being rhetorical.

A picture, on this site, is a projection from a point onto a surface, and it is correct from one place. The whole argument of the site rests on that: the focal length scaled to the display width gives the distance the reader must be at, and the number is a fact about the picture.

A 360° image has no such number. There is no place to stand from which the rectangle is a correct projection of the room, because the room is in every direction and the reader has one face. Ask the site’s own machinery for the viewing distance of a 360° picture and there is nothing to compute — a viewing distance is the focal length in the reader’s units, and a surface that has to be re-projected before it can be looked at has no focal length of its own.

What it does have is every ray. Given the file and a direction and a field of view, a correct flat picture can be produced, and that picture has a viewing distance and the whole site applies to it. So the honest description is that a 360° image is not a picture but a record from which pictures can be cut, and the cutting is where the projection actually happens.

This is the same distinction the essay on fisheyes past 180° reaches from another direction, and it is not a technicality. It explains why a spherical photograph shown flat looks like nothing anybody would take, why a viewer application feels necessary rather than optional, and why the interesting decisions about a 360° photograph are all made after it is taken.

What the other formats trade instead #

Equirectangular is not the only way a sphere gets stored, and the alternatives are worth setting out, because each one is a different answer to the same addressing question and each shows what equirectangular is buying.

The cube map stores six flat pictures, one per face of a cube around the eye. Every face is an ordinary rectilinear picture at 90° across, so every face keeps straight lines straight and every face has an honest viewing distance. Its distortion is bounded — the worst is at a cube corner, √2 focal lengths from that face’s centre, giving an area scale of 5.20 — where equirectangular’s is unbounded at the poles. The price is the lookup: finding a ray means deciding which face it hits, which is a branch, then a division. And the six faces meet along twelve edges, so there are twelve seams instead of one, and every filtering operation has to know about all of them.

The equal-area sphere partitions — the various schemes that divide the sphere into patches of equal solid angle — store every direction at the same resolution, which is the thing equirectangular most conspicuously fails at. They are used in astronomy for exactly that reason. Their addressing is arithmetic rather than geometric and does not reduce to two multiplications, so nothing that has to feed a display at sixty frames a second uses them.

The fisheye pair — two circular hemispheres side by side, which is what the sensor actually saw — needs no re-projection at capture at all, and preserves whatever the lenses preserved. It is the format with the least processing between the world and the file. It is also the one that makes every subsequent operation hardest, because the two halves are in different coordinate systems and the join between them is a circle rather than a line.

Set out like that, the pattern is the same one that runs through the whole field. Each format preserves one thing and destroys the others, and the thing preserved is chosen by the operation that will be performed most often. For a spherical photograph that operation is ray in, colour out, several million times a second, and equirectangular wins it by a wide margin.

What a view cut from it inherits #

Cutting a flat view out of an equirectangular file is a re-projection, and re-projection is lossy in a way worth stating because it is invisible in the result.

The stored file has a fixed number of pixels per unit of azimuth and elevation. A view cut looking at the horizon takes its pixels from the densely-sampled equatorial band and comes out sharp. A view cut looking straight up takes its pixels from the polar region, where thousands of file pixels describe one direction — so the view is built from a handful of distinct colours, smoothly interpolated, and is soft in a way no amount of resolution in the output fixes.

The consequence for anyone capturing is direct: point the seam and the poles at whatever matters least. That is the same class of advice as choosing where a map projection’s distortion goes, and it comes from the same arithmetic.

There is a second inheritance, harder to see. Whatever surface the original lenses used — almost always a pair of fisheyes near the equal-area rule — the stitch re-projects them into equirectangular, and the re-projection cannot recover what the lens did not resolve. A view cut from the file is a re-projection of a re-projection, and the shape it shows was determined by the first one. Conformality restored in software does not recover a shape the sensor never recorded.

The one point that has to hold #

Everything above assumes something the format cannot check and the capture usually gets wrong: that all the rays came from one point.

A spherical camera with two fisheye lenses has two entrance pupils, a few centimetres apart. The stitch treats the two hemispheres as halves of one sphere of directions, which they are not — they are two views from two nearby points. For anything far away the difference is negligible; for anything near the seam it is not, and the result is the doubled railing and the kinked lamp-post that anyone who has looked closely at the join of a 360° photograph has seen.

The arithmetic is the same as for a rotated panorama and it is worth having in numbers. Two centres five centimetres apart looking at an object two metres away disagree about its direction by 1.4°. On a 4096-pixel-wide equirectangular file, where 360° spans 4096 pixels, 1.4° is sixteen pixels. Sixteen pixels of disagreement is a visible ghost, and it cannot be stitched away, because there is no single spherical picture that both hemispheres are parts of.

What makes this the format’s problem rather than the camera’s is that the file has no way to say so. An equirectangular rectangle asserts, by being one rectangle, that its contents are the directions from one point. When they are not, nothing in the format records it, and every subsequent operation — every view cut, every measurement, every re-projection — proceeds on a premise that is false near the seam and true everywhere else.

This site’s whole apparatus depends on the same premise and states it everywhere: a picture is a projection from a point. The 360° format is where the premise is most often quietly untrue, and where the untruth is most invisible.

The seam #

One more property belongs to this surface alone and is a genuine nuisance rather than a distortion.

The azimuth coordinate wraps. The left edge of the rectangle and the right edge are the same directions, so the image is topologically a cylinder rather than a rectangle, and any operation that treats it as a rectangle produces a visible vertical line down the middle of the back of the scene. Blurring, sharpening, compression block boundaries, and every convolution written for ordinary images all do this unless they are told the width is periodic.

The two edges of the elevation coordinate do not wrap; they collapse. So the top and bottom rows are degenerate in a different way from the sides, and an operation correct at the seam can still be wrong at the poles.

This is the ordinary price of using a rectangle to hold a sphere, and it is the same price a cartographer pays at the antimeridian. It is worth naming because it is the one defect of the format that is not a trade — the distortion is bought deliberately in exchange for cheap addressing, and the seam is bought accidentally in exchange for nothing.

The summary, which is uncomfortable #

The format that stores nearly every spherical photograph ever taken is the worst of six surfaces on every property a picture surface can have, and it is the correct choice.

It is correct because the property that mattered was not on the list. The list — straightness, shape, area — describes what a surface preserves of the world. Equirectangular was chosen for what it preserves of the arithmetic: a direction goes to a pixel by two multiplications, and nothing else in the family comes close.

That is a genuinely uncomfortable result for a site whose habit is to measure what a projection destroys, and it is worth leaving uncomfortable. Not every choice about a picture is a choice about the picture. Sometimes the surface is chosen by the machine that reads it, and the reader inherits a distortion that nobody looking at anything ever decided to accept.