Not a native speaker, but I believe it could mean both. Certainly it can be about size (a big, large son) or fame (a notable, great son – perhaps it would be the default understanding without context, comparing to eg. duine mòr a great man in Dwelly). I cannot quickly find any examples in dictionaries that would suggest an adult, grown-up son, but I think it could used that way in context too, especially since a’ fàs mòr can mean growing up (lit. growing big, also becoming tall, big), but also a’ fàs suas would be the default for growing up.