In the last 30 years or so, in England, it seems to have become accepted that the person "in the chair" will be referred to as "the Chair". This used to get the response from old fogies (like myself), "I am not a piece of furniture", but no one seems to mind the committee being called "the Board" which obviously means "the Table". It's a lot better than using "Chairperson" if you want to be inclusive.
I agree, nor can you in the other Germanic languages I know of (including my native: Dutch). However, "chairman of the committee" is perfectly fine English (they use the genitive -s less than, say in Swedish, German or Dutch), and the translation of that to "kommitténs ordförande" seems right. Or would you translate that in a different way?