«Имя» is accusative (and nominative), «и́мени» is genitive (and dative, and prepositional).
When the verb is negated, and the object is not a living thing, you can use both accusative and genitive forms for the object.
The exact guidelines for choosing the correct form are quite complex. You don't really need to learn them (you'll get the patterns as you progress through the course), but if you're interested, you could look at my comment here: https://www.duolingo.com/comment/11633175 (abandon all hope, ye who enter here!)
Actually, «Я не зна́ю и́мени её ма́тери» is also not the most common way of saying it in Russian. It sounds somehow formal.
I think "I do not know what her mother's name is" is closer to „Я не зна́ю, как зову́т её ма́ть/ма́му» in Russian.