So the मेरे being in plural is because of the बेटे, not the भाई? That seems a bit confusing to me... Also, what does bhai-on mean? I thought bhai was the same in singular and plural?
भाई is the same in both singular and plural in the nominative/direct case. However, in this sentence, के requires it to be in the oblique case, in which भाई is singular, but भाइयों is plural.
मेरे भाई के बेटे = my brother's sons
मेरे भाइयों के बेटे = my brothers' sons
मेरे is oblique (not plural) because of the के, not because of the भाई. के is a postposition, and generally everything touched by a postposition has to go oblique.
The existence of bhai-on has never been mentioned so far in this course, so I, and probably many others, wrote the answer as "brothers' "