Yes, "ręce" is the Nominative (and Accusative) plural. But the sentence uses Locative (it's 'in your hand', after all). "ręce" is the Locative form of "one hand". For plural hands, it would be "Co trzymasz w rękach?"
Happened twice on this question: marked incorrect even though what i typed is exactly the same as the correct answer
Is it possible to say 'What are you holding in your arm?' I do not really know the difference between dłon (hand) and ręka (hand, arm?) Someone who can explain?
I believe that's technically possible but very, very unusual... you rather talk about holding something in your arms, plural.
Generally, "dłoń" is the inside of the hand (so the palm of the hand), but people use it for the whole hand, from the wrist. "ręka" technically is the hand, but in fact people use it for the whole limb.