Maybe it'd because I am rather deaf, but it sounds to me more like "Ydych chi'n gallu nofio gyda fi?"
I think it's as it should be
I have just listened to the audio above and I agree that now it sounds perfectly OK. I don't understand why it sounded so different when I did the exercise.
If you haven't been learning for that long it may just be that your ear is becoming more attuned to the sounds. The "h" of "hi" is quite aspirated, so it's not so strange that it sounded like a "ch"
Yes, it's probably that -- though I have been learning on and off for donkey's years! Maybe my heading aids are better adjusted today! I think I had better stop commenting on pronunciation!
Could this also be interpreted as asking permission? (e.g. "I like her. Can she swim with me, please?")
gallu/medru - being able to, having the skill/knowledge to
cael - being allowed to (and several other meanings and uses)
Your suggestion would translate as:-
Dw i'n ei hoffi (or 'hoffi hi). Ydy hi'n cael nofio gyda fi (efo mi), os gweli di'n dda (plîs).