Because go and gur correspond to "that" (present and past tense, respectively) while raibh is a verb (corresponding to snámh). If it were "they said that they were swimming," it would be dúirt siad gur raibh ag snámh. I think. :)
The FGB says that gur "Replaces go before past tense of regular verbs", so yes, it's go raibh siad, but gur shnámh siad or gur chan siad (that they sang) or gur ól siad (note - d'ól siad for "they drank" but gur ól siad for "that they drank").