Because of the capital letter V in Várba, they don't just go to 'the castle', but to 'THE castle'. And in Hungary, 'THE castle' is Buda castle. Similarly, in Czech, if you would capitalize the letter H in hrad (castle), you would mean Prague castle or even more specifically, the office of the president of the Czech Republic, located in the castle complex. It is really tricky though, and an explanation should be provided here.
It's an interesting point. I know from my own experience that that is exactly what happens when you talk about landmarks in a foreign language in real life. On the other hand, I guess we have to suspend our disbelief a bit and remember that the point here is to actually learn that "vár" is the Hungarian word for "castle", which is something I didn't know until literally just now, hahahaha.