"hos oss" ("hos" in general) specifically indicates that this is happening at our place our home, or, on this sentence, at the place of people the speaker addressess to. "Med oss/dere/ham/henne..." could mean anything. That they have a guest with them in the park or in the restaurant.
hos is a preposition that could roughly translate to at the place of or at the place belonging to. So when you say hos dere you're saying at the place belonging to you, so that's why it's just a pronoun an not a possessive pronoun. It's just that when you translate it to English, it works much better to say at your place.