One can think of it as the distinction between "to speak" and "to talk". As in English, there are both similarities and differences between the two.
Here are a couple webpages that might help:
In German, we prefer to talk "with someone" - it feels more like a conversation than talking at someone or "to someone" which sounds more as if you're just throwing your words at the person and not caring about what they say :)
Er spricht zu uns brings up in my mind a public speaker - who speaks "to the crowd" but not with them, not an actual conversation.