Each main color corresponds to three specific letters. For example...
z r k (blue)
b y d (white)
With the exception of just a few colors such as بني (brown) etc..explained last.
Each of those colors have masculine and feminine genders assigned to them based on the location of the three letters.
Masculine: 2 a _ _ a _
Feminine: _ a _ _ a 2
Fill in the blanks with the three letters from a color to observe its masculine or feminine form.
Furthermore, the gender of a color is determined by the object it applies to when used as an adjective, not the person you are speaking to.
As for the other colors, like brown (بني), simply use "ة" to make the feminine form of that color:
Masculine: بني (brown)
Feminine: بنية (brown)
I hope that helps.