Thinking about this further, I think Evangeline is right to some extent. If I use "my family comes from", I feel it places more emphasis on the multiple individuals within the family, rather than the family as a single unit. I wouldn't usually use it, but I think it's probably OK.
Yeah, English has the oddity of singular and plural verbs, and to top it off, they get irregular too. Generally if the verb ends in an S then it likely is a singular verb. And Family in the sentence above, is a singular unit. But remove that S and the verb becomes plural "They all come from…
Much of this goes out the window when using verbs in the first (& second) person: I come from… and some verbs don't obey this "rule" at all.
English is weird.
Why must it be accepted? In English, you could say "The family of my father comes from Norway", but without "the" it would not be correct. Also, it would be very rare for a fluent English speaker to say this. The idiomatic way to say it in English is to use "dad's" or "father's".