The « m' » is the indirect object of the verb. Both direct and indirect object pronounced are placed before the conjugated verb. In this case, it is the auxiliary verb "avoir" used in the compound tense.
The object of the verb 'reply' is what you say in your reply e.g. He replied yes. You have to have the word 'to' to show you are talking about the other person. 'He has not replied to me' is correct because 'He has not replied me' would mean 'He has not said "me" in reply'.