If you want to emphasize "any" in "I don't have any sister", you could say "У мене немає жодної сестри". Ніякої sounds like "not any kind of", like @va-diim wrote, sounds strange.
Yes, it's wrong. The object is negated, so its case is equal to the genitive case. The word "sister," сестра, declines to сестри in the negated, genitive case. The word "sisters," сестри, declines to сестер in the negated, genitive case. So your sentence would be, "Ні, у мене немає сестер."