The article must always go with the noun for each season as far as I'm aware. :)
http://www.dutchgrammar.com/en/?n=NounsAndArticles.07 This may help :)
I think that the ultimate sign of mastering a language is that you know when and where you can be sloppy in pronunciation. Which can be quite unexpected. For example: many people say pinnakaas instead of pindakaas (peanut butter).
But in this word, I do hear the f in herfst, and I don't know anyone who silences the f at all. You gotta find some way to pronounce all those consonants, however, so it may sound a bit mumbly ;)