дом = a home or a house, as the subject of a sentence. Example: My home is far away = Мой дом далеко
дома = (at) home, an adverb telling where someone or something is. Example: I am home = Я дома
в доме = in the house, or in a house (this form is always used with a preposition). Example: The cat is in the house = Кошка в доме
It's noun cases in russian. Дом is nominative and accusative case form, до́ма genitive, до́ме preposotional. The common meanings of that case forms are already explained in this thread by DyedBison.
The entire set of cases of a noun could be found in wiktionary https://ru.m.wiktionary.org/wiki/%D0%B4%D0%BE%D0%BC