Translation:I calculated wrongly, so I miscalculated.
It's usually clearest to ask about an entire sentence.
"Why would miss be incorrect for mise?" sounds to me as if you wanted to replace mi mise kalkulis with mi miss kalkulis.
Now I think that you are asking about an English sentence of "I calculated miss, so I miscalculated". Or maybe "I miss calculated". I have no idea because you didn't tell us more.
Either way, that would not work because "miss" is, as you said, a synonym for "mistake, blunder, etc." -- that is, a noun.
You can't say "I mistake calculated" or "I blunder calculated" or "I error calculated", so you can't say "I miss calculated".
Esperanto mise is an adverb -- so it can stand before a verb and describe how the action was done.
"wrongly" or "incorrectly" would be adverbs in English that could translate that.
We commonly say "miss calculated", I think miss would be an adverb when used in this manner. That is why I think it should be accepted.
Is it correct grammar to say "I wrongly calculated"?
To me, that sounds... wrong.