I would guess that it is likely fairly arbitrary. If you said that to a French person, I would guess that you would be able to express what you mean well enough. DL seems to split hairs a lot. I put "I have a fear of going there" and was marked incorrect even though the sentence expresses the exact same idea as "I am afraid to go there." Certainly there could be someone (including myself) who could argue against this, but, again, it would be splitting hairs and would be an arbitrary difference.
The French say "avoir peur". The English say "to be afraid". Try to translate the idiomatic French into idiomatic English: J'ai peur d'y aller => I'm afraid to go. It is not at all arbitrary and it isn't splitting hairs. You want to speak and write French correctly and you want to use English to translate it, not Franglish (literal translations of French idioms into awkward English).
I agree with Jacob. I think that the mere fact that french requires you to identify where it is you are going makes this sentence correct. If Y=there, so why isnt this "i am afraid to go there"? Or even "i have fear of going there." The latter may sound a bit awkward, but it gets the same point across - having fear of going to that location.
I put "I fear to go there" but was marked wrong. The correct translation is "I am afraid to go," which is pretty darn close to what I wrote except for the "y," which I translated because I thought I must. Anyhow, is my error that I translated "j'ai peur" as "I fear"? Since "peur" is not a verb, is the correct translation of "avoir peur de" always some iteration of being afraid, rather than fearing?
Actually, you have answered your own question. A sentence using the verb "fear" would use the French "craindre". Je crains d'y aller. In the expression "J'ai peur de...", peur is a noun and the French idiom breaks down (literally) to "I have fear of" but will be translated as "I'm afraid of...."