"My little brother is bad at swimming."
I'm definitely not an advanced Japanese speaker but I'm pretty sure what precedes が is the subject here. The の is there to make a noun of the verb "to swim" so that it can be a subject. The は is not a subject marker, it's a topic market and can replace or be combined with many particles. I don't know if there's any rule stating it HAS to be the first particle in the sentence but it sure very often is. So to break down this sentence literally (for the sake of my explanation, not for practical proposes) my coldest guess would be: Regarding my younger brother, his swimming is very bad.
Not to make things confusing, but が can sometimes mark what is considered to be the object in the English sentence.
The Particle Ga が II: The Object Marker Ga が
"In English, objects are typically linked to verbs of activity, but this is not the case for stative-transitive verbs. In fact, these verbs share much in common with adjectives. After all, adjectives are primarily used in expressing the state/condition of something...In Japanese, the objects of these so-called “stative-transitive predicates” are marked by ga が rather than wo を. "
- I’m good at math.
- I’m bad at physics.
"Stative-Transitive Predicates of Subjective Emotions"