Whose brother doesn't she like?
Her own? Then use "sia".
Some other girl's brother? Then use "ŝia".
The most likely interpretation of the English sentence is that it's her own brother that she doesn't like, so "Mia filino ne ŝatas sian fraton".
But without context, it's also possible that your daughter (Jane) doesn't like her friend Lucy's brother -- Mia filino ne ŝatas ŝian fraton.
Or what about this relatively plausible example:
- Ĉu vi konas Sofian? Alica ne amas sian fraton, sed Sofia ja amas lin.
- Ĉu vi konas Sofian? Alica ne amas ŝian fraton, sed Sofia ja amas lin.
Both translate to:
"Do you know Sofia? Alica doesn't love her brother, but Sofia does certainly love him."
But in sentence 1 it's about Alica's own brother (sia refers to something belonging to the grammatical subject), and in sentence 2 it's about Sofia's brother (ŝia refers to something belonging to another person than the grammatical subject)