In choosing my final answer ("We go to the back..."), I imagined something like "Burglary" or sneaking in through the back/rear entrance...
more literally "hátra megyünk" and far from "visszamegyünk" (although both are valid without "to the", depending on context)
For this exercise, I changed my answer to better suit the Hungarian circumstances, but in colloquial English one could definitely say "we go back behind those houses", not necessarily in the sense of "returning," but related to "we go forward up to those trees".
Surely a better translation to English would be "We go backwards, behind those houses. Nobody would say "we go to the back" in English unless they meant "we go to the back of the houses" and that would not be "hátra"