Would it be correct to use "za" instead of "zbyt" in this context?
Yes. They are not always interchangeable and it's not that easy to see the borders of when they are and when they are not; but it would work here without any problems and it is accepted.
I wrote "Czy jest ten płaszcz za krótki". How bad is it to change the word order?
Yeah, this sounds rather strange. As it is the overcoat that is too short, "jest" should stay between them.
How come when I forgot the jest it was marked wrong?
My guess is as good as yours, but I'd say it's because you forgot the verb.