Translation:Why do you not want to visit your grandmother?
The problem is, that "swój" was introduced too late in the course and people got accustomed to the 'normal' versions. So no, "twojej" is not wrong, but it is indeed less natural than "swojej".
Basically if only you are sure that a form of "swój" is correct, than it is better to use it. The 'naturalness' of a sentence is a matter of personal view. In my opinion, using "twój" when "swój" could be used is the worst, other pronouns somehow don't cause such a strong reaction in me.