No, it shouldn't. But many learners have problems with perceiving the difference between them. While to a native, the sounds don't even sound similar...
https://en.wikipedia.org/wiki/Close-mid_central_unrounded_vowel - this is the Wikipedia article on Polish Y, maybe it can be helpful.
Actually "mum" is used in most of the English-speaking European side, I believe... For "mam", Wiktionary says "Used in place of mum or ma in Scotland, Northumbrian dialects such as Geordie, as well as throughout Ireland and Liverpool, and the South Wales valleys; the Welsh word for mother is mam." Anyway, we can add it.