How do we know that it should be "We" when there is no "אנחנו" at the beginning of the sentence?
You are right, there is no way to know in this sentence. In fact, I would understand this sentence as "nobody talks about their clothes". In Hebrew we have a type of sentence called משפט סתמי, which is a sentence with no particular subject. It can be translated using the English "one" (one does not talk about their clothes), the German "man" or Spanish "se". It's a bit similar to passive sentences in English, for example:
לא רואים מכאן את המגדל -The tower cannot be seen from here
or - one cannot see the tower from here.
(The Hebrew passive, less frequently used sentence would be המגדל אינו נראה מכאן)