Translation:We used to come to this restaurant often.
Well, it depends on where the speaker is at the time. If he is in the restaurant, then it would be, "We used to come...." If the speaker is in another location, talking about that restaurant, then he would say, "We used to go...." Also, a French teacher of French language told me this:
Je trouve qu’il est mieux de dire « à ce restaurant »
Dans le restaurant! C’est évident que l’on est à l’intérieur quand on va au restaurant, ou à ce restaurant.
"Venir" is not a regular IR verb. So it doesn't follow the same pattern as regular IR verbs like "finir" and "choisir" in any conjugation.
Here's a full conjugation list: https://www.wordreference.com/conj/FrVerbs.aspx?v=venir
The same applies to "tenir", "devenir", "retenir" and other variants.