No, it is not necessarily the first noun. But you did not only switch the two nouns. You also detached the article "the" from "referees/judges" and attached it to "teachers". And that is what reverses the relationship, just like in English.
"A bírók tanárok" - "Tanárok a bírók"
"The referees are teachers" - "Teachers are the referees"
These two are more or less the same. Logically, all of the referees are teachers, but some teachers are not referees.
And with the transplanted article:
"Bírók a tanárok" - "A tanárok bírók"
"Referees are the teachers" - "The teachers are referees"
Here, all of the teachers are referees, but some referees are not teachers.
The word order is not even close to free. It is just more flexible than English, because all the important details that define the role of a word are actually glued to the words as prefixes and suffixes, in most cases. So you have more freedom in moving the words around.