Dw i'n mynd i nofio bedair waith y wythnos. Would that be correct for "I go swimming four times a week"?
Very nearly - no mutation after pedwar/pedair, that's all.
Why is ddwywaith mutated?
Because dwywaith y mis is being used as an adverb of time, describing how often you go swimming.
Similarly with, say, Dw i'n mynd yno bob dydd (I go there every day) - the first letter of pob dydd gets a soft mutation.