Shouldn't 别 work in place of 不要？
Yes, 别 is perfectly acceptable. You could also say "别睡了!", meaning either "Don't fall asleep!" or "Wake up!" depending on the context, more commonly the latter.
why is that that 觉 is jue (/) and jiao () ? is there a mistake here?
It's just one of many characters with multiple meaning and pronounciation. So context is key.
Please, Duolingo, get someone to review the accepted answers and stop failing people who give correct ones.
As I understand it, 别睡觉 should be accepted - in fact, 别 and 不要 should be thought of as interchangeable.