since the english says TO the apartment rather than IN the apartment, it seems it should be "k bytu" rather than "DO bytu"
But the meaning is important. "k bytu" means somewhere close to the apartment while the English sentence is normally understood he goes inside to visit.
This one should be also possible: Ten muž často chodí k mé matce do bytu
It is a different sentence. We reqwuire quite a strict adherence to the original here.