«Имя» is accusative (and nominative), «и́мени» is genitive (and dative, and prepositional).
When the verb is negated, and the object is not a living thing, you can use both accusative and genitive forms for the object.
The exact guidelines for choosing the correct form are quite complex. You don't really need to learn them (you'll get the patterns as you progress through the course), but if you're interested, you could look at my comment here: https://www.duolingo.com/comment/11633175 (abandon all hope, ye who enter here!)
I think a more common way to express this thought would be Я не знаю, как зовут её мать.
«Мать» is nominative singular (or accusative singular), «ма́тери» is genitive singular or nominative plural.
The word «мать» is irregular: it gets -ер- appended in most cases. You can see its declination in Wiktionary (click on the 'Declension of ма́ть' to see all the forms).
Слово "мать" в принципе редко употребляется в разговорной речи и звучит грубовато, разве нет? :)
Согласен. По-моему оно придаёт какой-то официальный оттенок.
"I do not know what her mother's name is" is an extremely common way of saying the same thing in American English - perhaps it could also be accepted as a correct answer?
Actually, «Я не зна́ю и́мени её ма́тери» is also not the most common way of saying it in Russian. It sounds somehow formal.
I think "I do not know what her mother's name is" is closer to „Я не зна́ю, как зову́т её ма́ть/ма́му» in Russian.