Would you consider adding "I did not reply to your question" which is more usual than " I did not answer to your question". If you wanted to use the latter then it is better to say " I did not answer your question".
Thanks Glavkos. To answer to means to be responsible to eg I don't have to answer to you, meaning you are not my boss I don't have to give you a justification. The second meaning is to answer to the cause, meaning to respond to an appeal. :-)