no, it was not yours just because he took it from you. you are seeing things that are not there. the dative means he took it from you, thus impacting you.
ETA: on second and third thought, the dative of involvement is so weird that it could have been your desk and/or your stapler, or even neither. if it is neither, i will need to think through the translations because the involvement would seem positive (gain) rather than negative (loss), like he brought/took it for you.
and yes, He took my stapler. is a good translation of Vzal mi sešívačku.