"Bye, we are going to grandma" or "Bye, we are going to grandmother" would sound odd in English if you're trying to say you are going to your grandma's home. This answer here is probably a literal rendering of the Russian.
every single other sentence in this lesson marked me wrong for translating идём (or other conjugations) as "going" and saying it's supposed to be "walking", so when I got to this sentence I finally remembered to say "walking" and it was marked as wrong and said it was supposed to be "going". Which is correct?
If you're going to see or visit someone (at home, at their office, etc.), then use "to" when you see к + a person (к врачу, к девушке). In this respect it's like В/На as "to" for physical locations (в кабинет, на стадион). You just wouldn't say в/на to talk about going to someone.
К would mean "towards" if you're specifically talking about approaching someone or something.
Мужчина шёл домой к жене - The man was going home to his wife.
Мужчина подошёл к женщине и попросил у неё сигарату - The man walked up to (approached) the woman and asked her for a cigarette.