The fruit is named orange because the sanskrit word नारङ्ग (nāraṅga) transfered through Persian and Arabic languages sounds like orenge to Old French people. The color name is based off of the fruit.
Similar colors before the word came were called things like saffron, or yellow-red.
I definitely think it should be pronounced "oransj". I have never heard a person say one single orange is "oransjE", I would only pronounce the "e" if I was talking about multiple oranges. I would say "Genseren er rød" (The sweater is red) and "Genserne er rødE" (The sweatERs are red).
The issue with "oransje" is that it's said differently, but spelled the same no matter what.