Man, ir and ser are so confusing!
Yep... in past they are very similar!!!
"My father was a castle yesterday" would be a surreal sentence, but I suppose that would be "foi um" not "foi a um"
How about "My father was at a castle yesterday"?
That is "meu pai estava em um (num) castelo ontem".