"to work at the restaurant" has to be translated as "im Restaurant arbeiten" (not only as a physical place, but as a workplace). If it would be "We meet at the restaurant", it could also be "Wir treffen uns am Restaurant", if you don't intend to meet inside. It's just a different usage of the prepositions, there is not one-to-one correspondence between them.