what has passé got to do with this?
It is a familiar idiom.
Alternatives: "où est-ce que tu étais ?" or "où est-ce que tu étais allé ?"
What "passé" implies is that probably the speaker had looked for that person who had somewhat "disappeared".
Is there anything grammatically wrong with "Où aviez-vous été?" as a translation of this, or is it just rejected because it's not the idiom?
Actually yes, "avoir été" is very often used as an incorrect replacement of "être allé".
In good French, you therefore should say "Où étiez-vous allé ?" and not "Où aviez-vous été ?".
It didn't accept tu?
I used tu as: ou est-ce que tu es passes? but it didn't accept it. what's wrong?
With tu (singular masculine or feminine): où étais-tu passé(e) ?
I've always wondered whether vous, when referring to a single person (politely), should still modify adjectives and participles to be plural. It appears that it doesn't.
That's right, conjugations remain identical to that of plural "vous", but since you address one person, adjectives have to be singular (feminine or masculine)
vous êtes gentil /gentille = singular
vous êtes gentils / gentilles = plural