Well, many sentences with a single item of clothing would indeed be better with Present Continuous, but so they would be in Polish. But it's not possible to change them that way.
This one doesn't even sound that bad, you (generally) wear a dress. Maybe even the same one too often. Who am I to judge.