Translation:You are going to the café with your brothers.
No, there is no translation for siblings, other than "frère(s) et soeur(s)".
"Une fratrie" is a group of siblings (all brothers or a mix) but etymologically it is based on "brothers", so we may not use it if sisters are majority, even less if there is no male in the group (as far as I know "sororie" for all sisters does not exist either).
Context and careful listening will help. "Vous frères" doesn't really make any sense here. But you knew that. So a careful listen will tell you. Remember, too, that people do not always pronounce words with pristine exactness. Some people speak very clearly and other people slur their speech, especially when speaking rapidly.
Certainly one may "go to coffee" although it is a colloquial expression used to mean "to go on a coffee break". However, the expression "aller au café" does not mean "go to coffee", but "go to the café". In many regions, a "café" is more commonly understood as a kind of restaurant or "coffee shop" where you may get coffee, but it is most often for having breakfast or lunch.