Yes, it would sound clumsy. Almost like saying my male sibling is swimming. One would simply say brother.
In English, the context usually gives the needed clues for gender ID of a cousin. You might begin the next sentence with the word "he," and then it becomes obvious.
Except in rare cases, no one ever says "male cousin" in English.