If I understand it right, -eusi=black, -eupe=white, and -ekundu=red are the only true colour words in Swahili. They act as adjectives and you have to make them agree with the noun by using the right prefix for that noun class.
All other colour terms are either borrowed from other languages (e.g. "pinki") or describe something in nature with a distinctive colour (e.g. "zambarau"=plum). These don't take a prefix; you use "rangi ya ..." (the colour of ...) instead.