Shouldn't "is madra agat" be acceptable if you're trying to say that they own the dog?
No. 'Is leatsa an madra' is 'you own the dog'.
"Tá" and "is" are used similarly but for very different purposes. Its hard to explain but "is madra agat" isn't really a real sentence. For comparison you can say "is maith liom" but you can't say "tá maith liom".
Would 'cú' not also be 'dog'? Would there be a difference between the two?
cú means "hound".
It is not a synonym for madra.