Do numbers have to agree with the classes? I'm confused because some numbers do and some don't.
Good question. Only the numbers 1, 2, 3, 4, 5 and 8 take agreements with the nouns they describe. All other numbers take no agreements.
The reason for this is that the aforementioned numbers are of Bantu origin and so inflect according to standard patterns. The remaining numbers are of Arabic origin, and so like most loan words are not inflected. So for 1 through 10:
-moja, -wili, -tatu, -nne, -tano, sita, saba, -nane, tisa, kumi