twojej/ swojej is often not needed. Regardless of a case. In Polish often if something is not specified it is assumed to be your own. It is a bit ambiguous. It could mean somebody else's grandmother. But assumption is it is your own.
In the popular book series the author forgot that one of minor character was established as orphan few books earlier. She had him say "Jadę do matki" (I'm going to (my) mother) , a book later she changed that to "Jadę do matki. Matki ucznia." (I'm going to a mother. A student's mother))
No, if you talk to someone you probably mean their grandmother not a grandmother of someone else. It should be clear from a context of conversation. If it is not clear, you can add some determiner - like for example "your". And in this sentence it would be "swojej" not "twojej".