Yes, children don't learn genders. They just gradually pick it up by hearing countless examples.
In Gaelic and Irish (more than any other language as far as I know) the majority are masculine. So assume masculine without evidence to the contrary. I don't want to give you all the rules at once, but the one that applies here is that words that end an i followed by a consonant are usually feminine.
Unfortunately this problem comes to frequently on this site. That dictionary was published in 1925 (even though it has been reprinted masquerading as much more modern). Worse, it was never accurate in the first place. Sadly its best use is for keeping warm in the winter. The best dictionary for simple look-ups is at faclair.com . The left column of results is modern and the right column is old but accurate.
The word is universally feminine as far as I know, and it would sound odd if a word ending in -ais were masculine. If you look in Wiktionary and follow the links back you find that it appears to have been feminine all the time and to have been of Celtic origin. However, it is alleged the word came to Gaelic via Scots, which would have caused the gender to be forgotten.
If you look carefully at the linked Wiktionary etymologies, you’ll see that the Gaelic word is a borrowing from Scots/English, in which breeks, breeches are native, descended directly from PIE *bʰreg- through Proto-Germanic *brōks.
The word seems to have been borrowed into Latin as braca and Gaulish brāca might have been a proxy there, or it might have existed natively in Gaulish parallel to Germanic (but Matasović does not reconstruct it for Proto-Celt.).
But anyway that form is not a direct ancestor of the modern Gaelic word (and couldn’t have been, unlenited g for one shows that it is later-than-Old Irish borrowing; also related words AFAIK are not attested in medieval Irish, edit: but much later 17th c.+ English borrowings briste and braitsi are, probably also Old Irish bróc, hence modern bròg shoe).
We read the same Wiktionary article but where you say 'it is the case', I say 'it appears to be the case'.
I just find it s bit suspicious that this word magically regains is feminine gender and acquires the matching slender s (that appears to come out of nowhere) unless there is some direct link somewhere with some language that has genders.
It’s possible that it is masc. to some speakers, but Dwelly, Am Faclair Beag, Colin Mark’s The Gaelic-English Dictionary, and Angus Watson’s Essential English-Gaelic Dictionary all agree on briogais being feminine.
And Maclennan’s dictionary seems to be pretty terrible (see the part under the The Dictionary You Probably Shouldn’t Buy header) and often misleading or inaccurate, so you might want to always cross-check it with Dwelly, AFB, and Colin Mark.