Why мій сім'я is incorrect ?
Because the pronoun "мій" is masculine while "сім'я" is feminine.
Why is family female and not plural?
I was wondering the same thing.
Well, because it is kind of a collective noun. My family is big - family is singular as well.