"I am visiting my grandfather."
Translation:Odwiedzam mojego dziadka.
Neither "mój dziadka" or "moja dziadka" make any sense, those are wrong forms.
The noun "dziadek" (grandpa) is masculine, so in Nominative (the basic form, mostly used for the subject of the sentence) it's "mój dziadek", as "mój" is the Nominative masculine form.
Now, the verb "odwiedzać" (to visit) takes a direct object in Accusative. Accusative of masculine nouns is the only situation in which it matters whether the noun is animate or not. Logically, "grandpa" is animate. The masculine animate form in Accusative is "mojego". Same ending will apply for other possessive pronouns and for adjectives.
See the first table here for reference: https://en.wiktionary.org/wiki/m%C3%B3j#Declension_2