this thing happens with personal pronouns only. You go verso di me, verso di lei, verso di noi, verso di loro. I can't think of the circumstances in which you can go "verso di te" but someone could throw something, a stone, "verso di te".
Instead you go verso il cavallo, verso piazza San Marco, verso Roma, verso New York and every other thing and living being.
I hope there aren't exceptions. There could be, because what I've just told you doesn't come from a grammar. I'm only trying to remember how I speak and creating a rule in this moment. I'm native, not a teacher.
"Towards and toward are prepositions. We can use both forms, but towards is much more common than toward. …"