Each main color corresponds to three specific letters. For example...
z r k (blue)
b y d (white)
With the exception of just a few colors such as بوني (brown) etc..explained last.
Each of those colors have masculine and feminine genders assigned to them based on the location of the three letters.
Masculine: a 2 _ _ a _
Feminine: _ a _ _ a 2
Fill in the blanks with the three letters from a color to observe its masculine or feminine form.
Furthermore, the gender of a color is determined by the object it applies to when used as an adjective, not the person you are speaking to.
As for the other colors, like brown (بوني), simply use "ة" to make the feminine form of that color:
Masculine: بوني (brown)
Feminine: بونية (brown)
I hope that helps.