The only solution I can come up with is to use transformer coupling with three windings - two input windings from the two sources and one output winding to the speaker. The windings could all be on one toroid. The only coupling would be magnetic. I haven't had the energy to calculate what values might work, but offer it as a conceptual solution.

Mike