"The neighbours invited us to look at their new TV set."
Translation:La najbaroj invitis nin rigardi ilian novan televidilon.
Here's how I would interpret some variations on the sentence.
"ili invitos nin rigardi ilian televidilon" -> the object of the invitation is looking at the tv
"ili invitos nin por rigardi ilian televidilon" -> they invited us (for dinner maybe) and the goal was for them to look at someone else's tv
"ili invitos nin por rigardi sian televidilon" -> they invited us (for dinner maybe) and the goal was for them to look at their tv (why? couldn't they do it without us?)
"ili invitos nin por ke ni rigardu ilian televidilon" -> they invited us (for dinner maybe) and the goal was for us to look at their tv
Yeah, no, http://bertilow.com/pmeg/gramatiko/pronomoj/si/simplaj_frazoj.html seems to imply it's perfectly sensible to use si to refer back to ili.
I got confused by this too. Apparently when ‘si’ is used with an infinitive it corresponds with the ‘senca subjekto’ of the infinitive. In this case that would be ‘ni’ which wouldn't make sense, so it has to be ‘ili’ to refer to someone else (in this case the neighbours).
But normally yes, ‘si’ can refer to someone in the plural.
Esperanto is hard!