In addition to what Almog wrote, גבוהה is pronounced differently to גבוה: gvoha versus gavo'a. And in fact, someone being very punctilious, such as when doing a Torah reading, will pronounce גבוה with an audible final "h" (gavoaH), it's one of the few cases ה is supposed to be pronounced as a consonant rather than a vowel at the end of a word.
Is adding "ה" to the end of a male word ending in "ה" to make a feminine word the exception or the rule? For example "גבוהה" has the added "ה" but "יפה" doesn't but uses nikkud and pronunciation to distinguish a male or female word.
As my previous comment hints, it's not an exception from a Biblical Hebrew point of view, since גבוה ends in a consonantal h rather than a vowel e, represented by the letter ה in יפה. Yafe is never yafeH, but gavoa is technically supposed to be gavoaH. It's like נוח noakh נוחה nokha, gavoaH gvoHa.
No, the first ה is a part of the stem, the second is the female marking. גבוה is the male form, הוא גבוה.
And it's not one of those things that gets simplified away in writing ("to the" is the lamed with a vowel rather than לה)?
No. There are a few 'stems' that have a final ה that is a consonant. About 5-6, and the others are not very common.