Remember that "a la iglesia" can also mean simply "to church," and in the same way, "a la universidad" can be translated as, "to the university/college" or simply, "to college."
Although it may be more clearly described as "a misa". Just as going to school is not the same as going to class.
To mass = a la misa.
To school = a la escuela.
In these prepositional phrases of both languages, the "mass" and the "school" are NOT "in general."
Habian can also be you plural, right? "you had gone to the church"
do you mean '¿(ustedes) habían ido a la iglesia?'.... yeah, it's right!