You're talking about the past, whereas the question is about the present. Imagine someone phoning you up and asking, "Have you gone to work?" - in other words are you there now, or are you on the way. What you wrote would refer to a conversation you have with the person the next day, about what they did yesterday or some other time in the past. However, in American English, Did you go to work would be an appropriate translation due to grammar differences.