"This is my sister whose boyfriend is a teacher."
Translation:Das ist meine Schwester, deren Freund Lehrer ist.
Because the subject with the boyfriend is female (= the sister = Schwester).
If we were talking about a man as subject, it'd be dessen:
This is my brother whose boyfriend / girlfriend is a teacher.
= Das ist mein Bruder, dessen Freund / Freundin Lehrer/in ist.