After eight, I would say. The numbers that take noun class agreements are -moja, -wili, -tatu. -nne, tano and -nane. So, the exceptions below ten are: sita, saba and tisa (because of their Arabic origin). Note that -wili becomes mbili for N/N nouns (and when counting, of course).
Also, please note that those numbers that take agreement will take them every time they appear as a unit, so, for example, 21 spoons would be vijiko ishirini na kimoja, and 2002 spoons would be vijiko elfu mbili na viwili.
That said, I have noticed that in speaking, a lot of Tanzanians will not do this, so they might order twelve eggs as mayai kumi na mbili.