Abergavenny doesn't make sense when you think about it.
Y Fenni is Welsh, and so is Abergavenny, so both names are in Welsh.
I'd say Y Fenni is more slang and Abergavenny is if you can be bothered to use that much breath
https://en.wikipedia.org/wiki/Abergavenny#Origins_of_the_town_and_its_name discusses the origin of the name.
Apparently, it used to be "Aber Gafenni" (which explains the English name), which then got shortened (in Welsh but not in English) to Y Fenni.