I'm going to argue that "I don't know whose mom it is" is the correct word order here. The word order of "whose mom it is" should be the same as in an affirmative statement, as this is the grammar rule in English for word order following phrases like "I don't know." e.g. We say "What time is it?' but: "I don't know what time it is."
In English, it can be either. Or "this is" as a 3rd alternative.
"It" does not refer directly to the person, but rather is a kind of token or place-holder loosely defining the situation (which is "not knowing"). This particular usage in English is far from being grammatically logical, and "she" would certainly be logical and clearly descriptive of the person - but speakers of American English use "it" like this all the time.
I would prefer to see "she" but suppose you are looking at a class photograph, which is an object, not a person. As stated above, "this is" is a third alternative, which I could see if one was pointing to a person in a photograph, and in this example is perhaps a "less contentious" choice than "it".
I am afraid that in this case "it" refers specifically to "mom". Thus, it really should be "she". The impersonal "it" is only correct when you can't find the noun it replaces. Such as talking about the the weather: It is raining. In this case "it" is not replacing a noun. But because English word orderis so fixed, there has to be a subject or a word where the subject should be. Think "there is". Although English is a very gender neutral language - or maybe because of it - we almost always use the gender specific pronoun when referring a person or something that has a gender - when we know what the gender is. When I talk about my dog, I always use "she". I hope I wasn't too long-winded.
This sentence, referring to a mother, is one of the very few instances in English where gender of a word REALLY matters. "I don't know whose mom she is" or possibly "I don't know whose mom this is" are acceptable translations. Referring to any woman as "it" is offensive.
That's just how it works. You'd use the same structure with the word "friend", which is clearly a he or a she, but uses "it" nonetheless: "Whose friend is IT?", with "it" referring to the friend. "Mom" and "friend" are both nouns that refer to people, but are referred to as "it" in this case.