Each color corresponds to three specific letters. For example...
z r k (blue)
b y d (white)
b n y (brown) etc..
Each color has masculine and feminine gender assigned to them based on the location of the three letters.
Masculine: a 2 _ _ a _
Feminine: _ a _ _ a 2
Fill in the blanks with the three letters from a color to observe its masculine or feminine form.
Furthermore, the gender of a color is determined by the object it applies to when used as an adjective, not the person you are speaking to.
I hope that helps.