Translation:The teacher is talking and we are listening to him.
"To" isn't referring to an indirect object here. It's just something we add after "Listen" for whatever reason.
An indirect object means there's a possibility for a direct object.
You can read a book to someone, or give a gift to someone, but you can't "Listen music to someone".
Ah yes, my mistake was not recognising what 'to' was actually supporting in the sentence, and in this case it's the verb, not the object. Of course there is no such usage of this grammatical function in Russian, so the circumstances can seem a bit blurred. Thanks for your help!