Good question. Only the numbers 1, 2, 3, 4, 5 and 8 take agreements with the nouns they describe. All other numbers take no agreements.
The reason for this is that the aforementioned numbers are of Bantu origin and so inflect according to standard patterns. The numbers 6, 7 and 9 are of Arabic origin, and so like most loan words are not inflected. Numbers 10 and above also do not agree with classes, regardless of word origin. So for 1 through 10:
-moja, -wili, -tatu, -nne, -tano, sita, saba, -nane, tisa, kumi