When use psa or pies?
Pies is the basic, Nominative form. "psa" is Accusative, which is probably the most widely used case. "widzieć" (to see) takes Accusative.
If you need more information about the cases, I'd recommend checking this topic for posts about cases.
Why "I am seeing a dog" is wrong?
That sounds like you're a bit too much into animals, if ya know what I'm sayin'
Is there a rule about which verbs take accusative, and which ones take instrumental ?
No, you have to remember them. Very few verbs in this course take the instrumental. Być, zajmować się, interesować się...
BTW 'psa' is one word and not pronounced 'ps a'.
Yes, that's probably the worst pronunciation in the entire course, but so far we don't have a possibility to change it.