We do know it, because there's always context for "swój". It's in the subject of the sentence (even if this subject is only visible from the form of the verb).
(Ja) lubię swoje biurko. = I like my desk.
(Ty) lubisz swoje biurko. = You like your desk.
(Ona) lubi swoje biurko. = She likes her desk.
(Oni) lubią swoje biurko. = They like their desk.
And so on, and so on. "swój" and its forms always refer to the subject of the sentence.
Not exactly. I mean... calling it 'reflexive' makes sense to me, but I believe reflexive pronouns are the ones like "myself", "yourself" etc. in English, so in Polish the only reflexive pronoun is "się".
Anyway, the important thing is to understand how it works, nevermind the names ;)