Three years since the above post, but never answered, and I got it wrong, too. It turns out that the preterite (simple past) conjugation for "ir" and "ser" are the same (not true for most of the other conjugations of the two verbs). Thus, one has to interpret based on context which to use. I think that, in this case, if Duo had wanted us to say, "Finally, we were at your office," the clue would've used "estar", i.e. "Finalmente, estuvimos a tu oficina."
I think it's a bit of a context thing. If we're saying "We couldn't find any place to put our stuff. In the end, we went to your office" Whereas this could say "We had a really drawn out meeting. Finally, we went to your office" Though I think you're right. If the word suggestion suggests "In the end" then that should be accepted.
Okay, bare with me for a second here. Is it possible that the reason "to go" and (this) "to be" use the same preterit conjugation is that from a meaning point of view, went and were, mean exactly the same thing? When a person says they went some place or was some place, they are both saying that there was some place in which they were present at that point in time. In fact, in the past tense "to go" and "to be" merge into on idea. That of being there. The meaning of "To go" in the preterit loses all sense of going, you were there period. We're confused because English ttys to make this idea of being some place in the past into two different idea. Is "I went there in July" and "I was there in July" really any different? In both cases you had to be there, and in both cases you had to get there. Just a thought.