This really threw me for a loop before I figured out what was happening because I assumed I had dodgy connections somewhere.
In the end I had to get a piece of software called 'Balance Lock'. It's not FOSS but it is free and according the app store doesn't collect data. It also has no ads or in-app purchases. I'm pretty happy with it so far. It does one thing and one thing well and that is to keep your audio balance at a chosen setting and shift it right back there again whenever that balance slider likes to go for a little walk. Weirdest issue, apparently been around for years.