? Doesn't "Er hört nicht" mean "He does not hear" rather than "He does not listen"?
It depends on the context. If this is about obedience or following instructions, "Er hört nicht" is fine. Otherwise, it should be "Er hört nicht zu".
Isn't there a better way to make this distinction in German? In English "He doesn't listen." carries very many different meanings than "He doesn't hear." German must have a way to make these differences too.
Why er hort zu nicht was not accepted?
That would need to be "Er hört nicht zu." Did you see Christian's comment above?
"er hört nicht auf" is also fine?
Zuhören = to listen to, aufhören = to stop. "Er hört nicht auf" = he does not stop.
Er hört nicht auf sie would be "he doesn't listen to her".