How it is not "bukan" because house is a noun?
Tidak is modifying besar in this sentence
This is the rule (I think) :
Tidak + verb, adjective
Bukan + noun.
How about Rumahnya?
That would mean the house or his/her house, but not my house.
Should "Rumahku bukan besar" be accepted?
I'm afraid not in this context. You can read demoksaputra's explanation above.
Is it wrong to answer "rumah saya tidak ada besar"?
Same question! When and when can you or can you not use "ada"? Is it only used for people?
ada means "there is" or "there are"
if you answer "rumah saya tidak ada besar", it would mean "my house there is not big"
so that's incorrect.