Actually, «Вас» in «Как Вас зовут» is Accusative, not Genitive. But «Вас» is the same in Accusative and in Genitive (unlike «мамы»/«маму»).
Literally it's "How do [they] call your mom?".
By not using a pronoun, and using a 3rd person plural verb form, you create an indefinite-personal sentence. Such sentences are used when the person doing the action is unknown and unimportant, and are often translated into English with the passive, hence "How is your mom called?" or, more idiomatically, "What is your mom's name?"
That would rather be "Как ты зовёшь свою маму?" or "Как вы зовёте свою маму?" My mom's name is Nadezhda, but I don't call her that, for sure :-)
The direct translation is "How do (they) call your mom?" But the person asking this question just wants to know her name, so it's better to use the most common English way to ask that.
In English, you would need to say "What's your mum called?" if you want to ask her name in that style. "How's she called"... is an adverbial expression, meaning with what means is she called. For example,"How do you call your mum in for lunch? Do you shout her name or do you sound the dinner gong?"