"I didn't respond to him" should also be accepted. Reported June 2018.
You are correct that "him" is the indirect object. However, "I" is the subject. There is no direct object in this sentence, but it's sort of implied in the verb itself. For instance, you can read a book or drive a car. "Book" and "car" are direct objects. But how can you "answer" (or not, in this case) a person? You "answer" someone with the thing you're saying:
I answered (something to) him.
I didn't answer (anything to) him.
I'm sure there's a term for this kind of verb that someone way smarter than I am would know. Apologies if this explanation is confusing.