Mostly yes. Technically it's not necessarily as much about how many people you're talking to as to how many people the glasses belong. For example, if you talk to Luigi about glasses that belong to Luigi and Marta, then "voi" refers to Luigi and Marta even though Marta is not present, and "I vostri bicchieri" refers to both their glasses even though you're only talking to one of them.
In some cases, you can figure it out from the context. For example, remember "bicchiere" is singular, and "bicchieri" is plural, and this often changes other words. "i vostri bicchiere" would be incorrect because it is the (masculine) plural for "i" and "vostri" but the singular for "bicchiere"--you'd have to use "il vostro bicchiere."