It is not wrong. "Gehen" means to go. "To go" can be go, leave, to do something, cheer someone on (word order inverted there) and others. Like lets say I run into my neighbor's yard and start screaming toward the window. He comes out and says "go!" i go home and say "she told me to leave." That go meant go (out of my yard) or leave (leave my yard) break it down, it IS in fact, the same thing. Different context.
Because ''have to'' is a generic answer english has, and it was accepted it for the same reason, though it does not mean the exact same thing as ''should'' or need. ''To need'' is ''brauchen'', and this specific sentence would be literally translated to ''shall we go'' but we all know that is rarely used so it accepts similar things.