"Brother, I want to talk to you."
"Frere, je veux vous parler" wasn't accepted. I get that maybe it's too formal for a brother - but is it grammatically wrong?
Surely, "Frère, je veux parler à toi" can work, too? That is literally, "Brother, I want to talk to you."
"Frère, je veux parler avec tu" doesn't work. Why does it have to be toi?
tu is a subject, toi is an object. It's exactly like the difference between "I" and "me" in english. You wouldn't say "Speak with I" , you'd say "speak with me".