I wouldn't say that the word “ghastly” implied a supernatural being. You can read it comparably often about people or officials being “aghast” about decisions they did not agree with or encounters that took place out of the sudden. In spite of the seeming relation between ghast and ghost, we could speak of a false cognate.
At least this is how I usually understood it, just my two cents.
EDIT: I saw that the word influenced the spelling of ghost, but still, the usage frequently refers to a feeling of fright etc: https://en.wiktionary.org/wiki/ghast
Touché. :D I would then opine that it might not be too strong to address the hideousness of a castle but it was simply displaced in this context. I couldn't even imagine saying that a castle was “looking ghastly”, even when speaking of a haunted castle in the Polish countryside.
This also remindes me of that I need to look up why the Czechs also have the word “zamek” in their vocabulary when they already have “hrad” for castles.
EDIT: I need to improve my vocabulary in English to distinguish these two as do the Polish people and tez Czech: Castle for hrad/zamek and château for zámek and whichever word, beside pałac, which is a loan word, Poles chose. My reference site mentions zámek again, but we cannot just brush over these two different architectural styles as if we didn't care, there's a difference between residing and manifesting a strategic defence.