Three years since the above post, but never answered, and I got it wrong, too. It turns out that the preterite (simple past) conjugation for "ir" and "ser" are the same (not true for most of the other conjugations of the two verbs). Thus, one has to interpret based on context which to use. I think that, in this case, if Duo had wanted us to say, "Finally, we were at your office," the clue would've used "estar", i.e. "Finalmente, estuvimos a tu oficina."
I think "In the end" should be a valid substitute for Finally. At the moment, it is not accepted.
I think it's a bit of a context thing. If we're saying "We couldn't find any place to put our stuff. In the end, we went to your office" Whereas this could say "We had a really drawn out meeting. Finally, we went to your office" Though I think you're right. If the word suggestion suggests "In the end" then that should be accepted.