there may be loads of subtleties that i don't notice, but in der Nähe uses nearness as a noun; something like in the vicinity. the others use it more like an adverb, or preposition... (I think) :)

I hadn't thought there was much difference but taking your first example I'd say:
Der Hund sitzt nahe am Haus. - That's very close to the wall.
Der Hund sitzt nahe beim Haus.- That's not that close, maybe some meters or more depending on the size of the property.
In any case, an is closer than bei.
Der Hund sitzt in der Nähe des Hauses. That can be quite distant, is not really close and depends heavily from the size of the area you are taking into account.