No. 'Whose' implies possession.
- "Whose family?" "My family"
'What' doesn't really make sense in this context; if it were for species it'd be okay.
- "What/which family?" "The empididae family" (insects)
Either way we don't use 'що' to mean that kinda of 'what'. We'd use 'що' for like: що це? - "what is this". The 'what' you're thinking of would translate better to 'which' (яка).