Just like with weather, 'hi' also refers to the general situation. This sentence transliterates to "Would it not be better with you swimming?".
Your suggestion only works with a slight alteration:- Fasai nofio ddim yn well 'i' chi? = wouldn't swimming be better 'for' you?
The construction above is the Welsh equivalent of expressing preference.
Basai hi'n well gyda fi(gen i) = I would prefer etc....
Here's a song with the chorus 'mae'n well gen i........y hogiau go iawn' (I prefer....the real boys)
Try to look at it this way - "would it not be preferrable to you to swim?"
It's not the exact same (and obviously, the prepositional usage differs between the languages), but it's a decent approximation to figure out the sentence construction. :)