Ah! You know, that's kind of interesting, because I noticed this "bouncing off to the right", in some cases, when I was experimenting with it last night (I'm not "up and running", at the moment). But it didn't really put me off... In fact, quite the opposite. If I closed my eyes, and tried to really imagine the players in front of me, it was alarmingly convincing. It seems to me that I noticed it more at specific pitches, so I just figured it was the acoustic effect we've been talking about. Do you notice it at all pitch ranges? Also, how wide (roughly) are you setting your stereo width?
I know this is going to sound terrible, but maybe you just need to leave it for a bit. You may be getting kind of obsessed with hearing it, and thus fine-tuning your ears to picking it up. The fact that this "bouncing to the right" happens repeatedly with the HO-4 could just as easily be proof that the effect is, in fact, acoustic, since the acoustic characteristics of the horn, combined with those of the room, would obviously be totally consistent (and thus always "bounce" to the right, as the sound decays).
As a little tangent to our conversation, I have a mild tinnitus, which I perceive in my right ear. If I suddenly "notice" it, it gets appreciably louder, in a kind of "ramp" - like I'm sliding up a fader. If I forget about it, I literally don't hear it anymore. I can even control its level, to some degree, if I really concentrate. I did some research into it, one day, and found that nobody really knows what it is, but that most researchers today feel it to be primarily psychological (or, perhaps more accurately, neurophsyiological) - and they don't generally link it to hearing loss, as they did once upon a time. It's mostly linked with fatigue, stress, and caffeine, these days. All of which would generally apply, in my case! ;-)
So, as glib as it sounds, I'd honestly recommend just narrowing your pan a bit, and leaving it, for the time being. I don't think it's your speakers, though it could be your room (not likely). But, for what it's worth, I do also hear this "bouncing to the right"... it's just that it doesn't really bother me. Rather, it makes me feel like I'm actually sat in the "Silent Stage" with them!
cheers,
J.