"Дома" is an adverb of location meaning "at home," while "в доме" means "in (the) house," with "доме" being the prepositional singular form of "дом" (house).
To further add the the confusion, the nominative/accusative plural of "дом" is "дома," the only difference being that the stress is on the "о" in the adverbial form and on the "а" in the nominative/accusative plural form.
The difference between the syllable stress in adverbial дóма and nominative/accusative (inanimate) plural домá should change the pronunciation of the o, correct? Adverbial Дома should be "DOH-mah" while plural дома should be "dah-MAH"??? Unstressed o pronounced like a, right?
I hear that kind of thing a lot in other languages, too. I think - and this is just my opinion - that when pronoun я, the speakers tongue is get ready to move towards pronouncing не. я by itself is voiced, with the tongue pulled back in the mouth, while не is unvoiced, with the tongue very much toward the front of the mouth. If the tongue is preparing to say не while saying я the tongue is more towards the front of the mouth, causing я to have more of the ye sound you're talking about.
Try experimenting by saying я and не separately, with the я well back in the mouth, then note how far forward the tongue moves to say не Gradually speed it up, but keep trying to say ya. You'll probably find that the contortions your tongue has to go through to get from the back of your mouth to the front become rather inconvenient, so that you actually move towards the ye ne sound naturally. It's a question of efficiency and speed more than anything else. In m opinion, that is.