Swój is used when the possesor is the same as the subject of the sentence. For example:
Rozmawiam ze swoim ojcem. = I am talking with my father.
On zgubił swój telefon. = He lost his phone.
Znalazłeś swoją sukienkę = You found your dress.
Note that in each of these sentences, the subject is the same as the possesor (he-his, you-your, I-my), so you use swój. In the sentence "Which dress is yours?", you is not the subject so using swój makes no sense.