From what i've found out; please do correct me if i've put misleading info.
- Vem (subject) frågade (verb) dig (objective pronoun), meaning who asked you (you as the object of the action).
- Vem (direct object) frågade (verb) du (subject), meaning that you asked someone that the third party does not know who it is and wants to find out.
It changes if "you" is object or subject, and hence the one doing the asking, or being the recipient of the asking.
In English as well as non-question Swedish the word order also changes, but you can still see the same form change (he/him) in e.g. "he asked Tom" vs "Tom asked him" in English.