«Чей этот дом?» would be a different sentence grammatically. «Чей э́тот дом?» is a question with an answer «Э́тот дом мой», «Чей э́то дом?» is a question with an answer «Это мой дом».
It's not a possessive adjective, it's the same «это» that is used in «Это мой дом». When asking a question, we move the question words to the beginning of the sentence, but we keep the grammatic structure.
It's awkward English, but I think a literal translation of this sentence is something like: "Do you know, whose (is) this, the house?".
Idiomatic English turns this into:"Do you know whose house this/it (is)?" If you re-translated that word-for-word back into Russian, then it would be: «Ты знаешь, чей дом это?» but I believe that would be odd Russian syntax (word-order).
Чей never means 'what?', it always means 'whose?'. Not just in this sentence, but anywhere.