Can CONVENIR be used here.....
Cette robe ne me convient pas bien.
Yes and no. It won't mean it's not your size, but rather it's not the kind of dress you like or need for that precise occasion.
Cette robe ne me va pas tres bien