Polish numerals are generally very difficult. Let's start from the beginning of the sentence: "mam" takes Accusative, so the noun phrase "trzy psy" (as a whole) is in Accusative. As dogs are 'not masculine-personal plural', the Accusative form is identical to Nominative, so at least it's easy here. One dog is just masculine and animate, so the form is identical to Genitive.
But if this sentence mentioned some 'masculine personal plural' noun, your form would be right. For example "Mam trzech synów" = I have three sons.
Also, the numerals themselves need different cases. If we were to consider just the Nominative forms ("n dogs" as the subject of the sentence), most numbers need Genitive. (5 psów, 16 psów, 3629 psów) The numbers that take Nominative are: 1, 2, 3, 4 and those ending with -2, -3, -4 (BUT NOT ending with -12, -13, -14 - those also take Genitive :/)