Translation:The relationship to my brother is not so good.
It could mean either. Usually the setting should tell you which, but it can also be made clearer with possessives:
"Forholdet mitt til min bror er ikke så bra."
"My relationship with my brother is not so good."
"Min brors forhold er ikke så bra."
"My brother's relationship is not so good".
One question: Why 'the relationship TO' and not 'relationship WITH'??!?! 'Relationship to' makes no sense...