I think the “more to it” might be significantly crazier than the timing thing.

Or ears have unique complex shapes that attenuate certain frequencies and bounce sound around in complex ways depending on the direction they ate coming from. And our brains instantly process all that stuff too. It’s why our sense of hearing isn’t just on a flat plane around our head.

My hearing is pretty severely damaged in my left ear, and for several months I thought everything was to my right. but my ability to locate sounds has come back. My hearings not any better, my brain just figured out that my left ears fucked and compensated.

Put on some halfway decent headphones and try out the virtual barbershop.

I got into an argument with someone once about this, when they told me (paraphrasing) "it's safe to drive listening to music through headphones, because they let outside sound in".

Yes they indeed might, but - even ignoring delay introduced from digital electronics - you've now lost all sense of where that sound is coming from, because you're listening to the sound of one microphone being played through one speaker.

The human ear really is an incredible thing.

You can also detect is the source up or down thanks to ear shape which delays sound for couple of ms.

That’s boring. Two ears only allow you to put the sound somewhere on a plane (the vertical one that cuts your body in half lengthwise). How do you know the ‘height’ of the sound on that plane? By utilizing the different distortions the sound goes through while being funneled through your auricle.