I try to avoid this problem / that problem... / it by using the quite awkward construction with "it". It is annoying that duolingo says I don't know "helst", when it is quite a different problem. So... "that" is much more common in English (in such a context)? I usually use "this"...
In terms of English usage, 'this' or 'that's depends on context, rather than one being more common than the other per se.
If I were to say "I would rather not do this", I'd probably be referring to something pretty immediate - I don't want to do this right now. If I used 'that' it's more likely to be something in the future, or an alternative ("I don't want to do this, but I don't want to do that either"), or something that someone else has suggested: "Shall we go for a walk?" "No, I would rather not do that."
I think there are more situations where 'that' is more appropriate than 'this'.
As Deliciae says, 'that' refers to things which are further away physically and emotionally, but also temporally.