This just strikes me as logically wrong. It's like the use of brackets and signs in arithmetic. Each sign affects the units (numbers/word) adjacent to it in a particular way. An exclamation or question mark, as you say, inflects the text with which it is logically associated, whether that is directly quoted speech or the sentence that reports it.

3 + (6 x 9) isn't the same as (3 + 6) x 9.

I think you could argue that in most cases common sense will indicate the correct interpretation for the reader, but in those examples of the exclamatory reporting of a question, or questioning reporting of an exclamation, it's important to ascribe each attitude correctly to reporter and reportee.

That said, I find - the the whole business of the conventions of punctuating reported speech rather messy.

I think I would prefer some system like a colon then new line, with no quotation marks at all, or perhaps a dash to introduce them, but not to close them, kind of combining the Spanish prose approach and a play or film script.

e.g.:

He urgently replied:

– Don't panic, I'm almost there.

And maybe we should just get rid of exclamation marks completely!!!!! They are more abused than useful, perhaps.