This sentence was pronounced with no liaison between elles and a. This makes it impossible to distinguish between singular (allait-elle) and plural. Unless there is a clear verbal clue, both should be considered correct. Duolingo users should not be required to be mind-readers.
The liaison between elles and un is optional (at best). Therefore, you are right both singular and plural versions should be accepted.
However, this sentence is wrong in the tense:
"allaient-elles à un château ?" = were they going to a castle?
"did they go to a castle?" = sont-elles allées à un château ?
A question on this though... "did they go" implies to me a specific event in the past in English, so... would this not take the past perfect in French? Because here the question in French is in the past imperfect tense - which to me in English implies "Were they going to a castle"? (that is, "Were they (in the process of) going to a castle?")