Agree. Both ‘with you’ and ‘on you’ are fine. ‘On you’ is more immediate and to hand than ‘with you’ and I guess comes from the longer version of ‘on your person’, which sounds old fashioned nowadays. If you answer ‘yes’ to the question ‘on you’ then we know that you have whatever the object is, e.g.. the pen, the money, the passport, on you, that is, on your person, in your pocket, in your hand. If you answer yes to ‘with you’ then we know that you have brought your passport with you, but you may not have it immediately to hand. However, it is only a subtle difference and both should be accepted.
If I heard that I would assume it was a non native speaking. What's "proper English"?