You could say that and usually would. The phrase החולצות של הגבר gives all the required information about the shirts. If you use the possessive suffix, חולצותיו, we don't know who הוא is. We are either assuming that the information is given in a previous sentence, or it has to be in this sentence, so there is a redundancy somewhere. The problem doesn't exist when the shirts are "mine" or "yours": חולצותיי for example doesn't require more information. Anyway we don't use possessive suffixes very much, only for very specific words.
It's a necessary thing to understand if you want to understand and be understood in Hebrew. It isn't something that will be said all the time, but it's used; in things like endearments, or for example, my Hebrew teacher always referred to her husband as בעלי, never once הבעל שלי. It isn't necessarily something you'll need to be able to produce a lot, but you definitely need to be able to recognise and understand it.