Translation:The relationship to my brother is not so good.
It could mean either. Usually the setting should tell you which, but it can also be made clearer with possessives:
"Forholdet mitt til min bror er ikke så bra."
"My relationship with my brother is not so good."
"Min brors forhold er ikke så bra."
"My brother's relationship is not so good".