You can use either. The "o" is there to indicate the gender of "pai", but it is implied in the word "seu". It is the same with O meu carro vs. Meu carro ("meu" can only mean mine, and indicates the gender of the word "carro") and Tua escola (your school), for example.
As it turns out, the use of a and o can be a little redundant, so I guess that's why we do not feel too bad skipping it. =)