The essential matrix does not determine a camera pair. It determines four, all of which reproject every correspondence exactly, and the thing that picks one is not more algebra — it is the assumption that the photographer could see what was photographed.
All ladders