Could be. But I guess that you would also say "Han hade fyra barn" if you are talking about someone you used to know and don't see anymore. It's probably the same in English.
I guess you could also think about it in the way of "He had four children, now he has 2 teenagers and the other 2 are grown." So he would still have all four "children" but they would not be the age you would refer to them as barn? At least, logically that's how I make sense of it.