I agree, 去回家 sounds like too many verbs.
However, sometimes double barreled verbs work. In this case 回去 is an option (remember hearing it) so just change the word order.
but I don’t actually know if the whole phrase 回去家 is something one would say or not.