I think this is weird, going "to" , as a question. It doesn't sound right.
(In this sentence, without me at the end, clearly there are many cases of going to... Where are you going to without me doesn't sound right). The "to" is superfluous.
English is not Latin. Though I would not end that phrase with "to", it is not improper grammar.
I think it sounds better to put the preposition first, like this: To where are you going without me...but maybe I am wrong.