I've added it as an alternative.

The keys don't belong to the house (they belong to you), nor are they part of the house. They are the "house keys" or the "keys to the house".

"keys of the house" is not natural English. The only references I could find for it were from the bible (the key of the house of David), and it doesn't mean a literal key.