I’ve an M1 Mac Studio with Bluetooth mouse, keyboard, AirPod and AirPod Pro, switching between APods whilst gaming, and they all work flawlessly.
It sounds like the keyboard or mouse is the issue. Since when they’re disconnected the problem is resolved.
Next time it happens try removing one of them, test the audio, and if the problem is resolved that’s the device messing up Bluetooth.