It doesn't accept "we visit family members", either.
I wonder if it's because Duo is using a non-standard definition of familiaris?
Lewis & Short only have it as an adjective, with meanings like "of or belonging to servants; of or belonging to a house, household, or family; familiar, intimate, friendly; of or belonging to one's self; fitting, appropriate; by families; familiarly, intimately". (Notice that there's not even a whiff of "relatives".)
Wiktionary does have it as a noun, but with meanings "a servant, domestic" and "a friend, a familiar acquaintance", and latin-dictionary.net elaborates on those: "familiar acquaintance/friend" and "member of household (family/servant/esp. slave)".