I see some people have a difficult time understanding "hos oss". I do not know what I am about to say is completely accurate, but if you also know French you can translate this phrase as "chez nous", for example.
If this is not your case, I imagine "hos" to be like saying "Hey, let's go to Joe's", not implying a specific building (house, room, appartment) but it does make sense and you know you are going where Joe is living. I hope this made-up rule helps.
Ah, that makes more sense! Swedish is difficult when you're mother tongue is Dutch, because they are so similar, yet so different. It took me weeks to get used to -en meaning 'the' and not plural, as it is in Dutch. In Dutch it's 'fiets' - 'fietsen' (bike - bikes). I had to reprogram my brain to not automatically see it as plural. /fun language fact
Tack för att förklara!
No, hos means "at one's place", or more specifically "at a place belonging to". So you can eat hos someone's home, or someone's restaurant, or someone's offices, etc. - but it does not mean "with". In fact, if I eat hos dig ("at your's"), it is assumed without context that you're also present, but it's not necessary.
Edit: There are a few fixed phrases where it can mean "with" as well, but it does not extend to generalities.