"mit Freude(n)" would be the word to use, but that sounds just about as high-level and impractical as "He swims with pleasure".

I think "gladly" is just Duo trying to find an English translation for "gern(e)", which, if you leave the "often" out, would probably be written as "He likes to swim / swimming" --> "Er schwimmt gern(e)". But with the "often" in there, you can't work with "he likes to do it" in English anymore.