What is wrong with ''Chcę mówić z lekarzem''?
Generally we use "(po)rozmawiać" for having a conversation, although I can imagine a native using your version. I think it's colloquial.
Seems quite impolite... Or doesn't it sound so to Polish ears?
It may sound a bit impolite, I guess. "[Chciałbym/Chciałabym] porozmawiać z lekarzem" would be better, but then of course that's "I would like", not "I want".