It would be okay in context, but it's a slighly different meaning. "Mit" could mean you have it somewhere in your luggage, or in your car, or you brought it with you to the country. "Bei dir" implies you have it on your person right at the moment. "Have you brought your passport with you?" would be closer to your tranlation.
You'd also see 'mit' in something like, "Bringst du dein Pass mit?" (Are you bringing your passport along?)
"Bei" is also used when you live with people. I wouldn't live 'mit' my parents, rather 'bei meinen Eltern'.
But like @sakasiru said, in context you would be understood if you said 'mit' as well.
It's really not one of the meanings of 'bei'. This particular phrase is difficult because it becomes very idiomatic. If I say I have my passport 'on me', I don't really mean that it's 'on' me (not like a hat is on my head), so we end up with idioms in both languages. 'With' is, I think, a better translation in this case: Do you have your passport with you?
It is not incorrect in one case. "You" can be left off when you are telling someone to do something. So the statement "Have your passport on you" is correct English, but it is an imperative rather than a question. Since the German sentence here is a question, that would not work.
You are probably used to writing the verb "to have" at the beginning of a sentence, but that's because this is used for questions with the perfect tense, in which case it is the auxiliary of another verb, that goes to the participle: "have you brought your passport?" In this case we have something different. It is the verb "to have" meaning "to possess", in the present tense. For making questions in the present tense, we always use the auxiliary "do": "do you have your passport?"
It just doesn't work at all, unfortunately. For many phrases, 'bei' does not usually translate to 'by' in English, but a lot of times has the idea of 'with' or 'near' (Ich wohne bei meinen Eltern = I live with my parents). Also, 'by yourself' means to be alone. What you want is to say 'Do you have your passport with you'.