I agree with you. As a native Hungarien speaker I translated it by the house and I think there is the same distinction in Hungarian and English in this case. I have reported the sentence.
So the sentence means that there are no workers gathering outside but close to the building and we don't know if there are any worker inside the house
As a native English speaker, I would agree that being at school always means in the vicinity of school, but does not necessarily mean inside the building. If the context is that the subject is a school-aged person or a teacher and the implied timing is during school hours, then at school could be understood as being in class and thus specifically inside the building.