"They are not in the house."
Translation:В доме их нет.
Not according to Duo. There's probably something idiomatic about the word order. Google Translate says that в доме их нет means "They do not exist in the house" = "They are not in the house". Reverso.net says it means "In the house they are absent".
Google says that в доме нет их = "The house does not have them". Reverso.net says that it means "In the house is absent they".
A context search at reverso.net on нет их return some phrases, none of which were about them not being somewhere, but a similar search on их нет returned multiple instances of them not being in some place.
So, it seems safe to conclude that нет их is wrong as a translation for the exercise sentence.
What case is их here, and why?
It's genitive. You use «нет» + Genitive to express absence.
What determines which one uses?
You use «них» after the preposition (unless the preposition refers to a noun after pronoun, and the pronoun modifies that noun, e.g. «у них», but «у их сы́на»).
Дома is a fixed expression for "being at home". It's not a declension of дом when used like that.
And it can be a declension of дом: genitive singular, nominative plural and accusative plural.
So, you would use дома as a declension eg. If your saying "they go to their houses": они идут в дома
Or "there's a restaurant at my house": У меня дома есть ресторан