"You" can never refer to "we" or "them". It's always the second person. There are, however, different forms of "you" depending on who you talk to. It can be masculine singular אותך (pronounced otchá), feminine singular אותך (pronounced otách), masculine plural אתכם (pronounced etchém) and feminine plural אתכן (pronounced etchén).
Also, in Hebrew there are only two genders, masculine and feminine, there is no neutral.