I agree. But these kinds of mismatches are common in English, and I must admit "the ages of my grandparents" does sound a bit clunky in informal speech. It would be interesting to find out from native speakers if las edades de mis abuelos sounds similarly awkward in Spanish.
I wish this speaker did not occasionally drop the "s" at the end of some words. Sometimes the context makes clear that the word is plural ("mis" in this sentence, even though it sounded to me like "mi"). But sometimes the context is not clear, and I'm unable to hear the speaker pronounce the final "s." Anyone else have this problem?
Reading this reminded me that we have a paternal grandfather and a maternal grandfather. So I hope to learn too whether 'abuelos' can mean one's 'grandfathers'. (It would be a legitimate sentence such as in a discussion about grandfathers in general.)
In your English sentence there needs to be an apostrophe: "grandfathers' ages".
The exact same ambiguity exists with padres, hermanos, niños, hijos, tíos, sobrinos, esposos, reyes, etc. and I have yet to find any universally accepted way of distinguishing them collectively.
So I usually just refer to them separately and specifically, eg la edad de mi abuelo paterno y mi abuelo materno.
Sometimes you can get away with male/female (varón/hembra) eg mis hijos varones (my male children) but it often provokes weird looks from the unsophisticated and doesn't work across family tree branches, eg mis abuelos varones comes across as my male grandparents---as if all four of them were male lol.