What is вашей, and why not ваша?
As a native English speaker this feels wooden to me. I'd say "Does your mom have any brothers?"
I would too...
"Has your mom got brothers?" - in England we can say that
Isn't it plural? I would have said. Do your moms have brothers?
It is probablu formal, isnt it?
It's neither plural nor formal. "Мамы" is the singular genitive here, not plural nomunative (they just happened to look the same for this word).
Why is есть necessary in this sentence?
You need "есть" when the existence of something needs to be established. If you omit "есть" here, it would mean it's already known the mom has some siblings and now you are inquiring about their gender.
Why is it Вашей?
That's how the singular genitive form of "ваша" (feminine version of "your") looks.
That's just how it is. English has nonstandard plurals too, right?