Translation:I wear summer clothes.
In general, I would say, that it's just something which developed over time when you use them as compounds an when not. However, there are some rules to the chaos.
In written texts, especially in newspaper/research work headlines etc, the nouns are for the most part directly connected. If you have many nouns after each other, the same rule applies to avoid many の in one sentence.
Your example is different, though. 牛乳の会社 would be a milk company, while 牛乳会社 would be the milk company (as in the name of the company itself).
Well; French Polynesia is in the southern hemisphere, and although they only have two seasons there (rainy season and dry season), they have "summer holidays" in July and call summer the same part of the year that people in the northern hemisphere do. Although it may be because of their connection to France and the influence of French media and administration.