keinen refers to one hat, it's like you're saying 'I don't like any one hat...of these hats'. So 'dieser Hüte' is genitive but 'keinen' is accusative. I know, it's weird.

It's correct but genitive is generally considered better style.

'Keinerlei dieser Hüte' is wrong. If you want to express 'none at all' in this sentence, you could use 'Ich mag überhaupt keinen dieser Hüte'. 'Keinerlei' implies that you are speaking about an unspecified set of hats, which contradicts the 'dieser'. Technically, 'ich mag keinerlei Hüte' would work. But normally, we don't use 'keinerlei' for concrete objects very often. It's more appropriate for abstract nouns like: 'keinerlei Anstrengungen' (no efforts at all) or 'keinerlei Verständnis' (no understanding at all). I wouldn't bother too much about this word since it's usually fine to use 'keine/keiner/keines' instead.

I had to look this up. "Keinerlei" is not used very frequently and means something like: non at all which would be too strong for the original phrase. A given example was: ".. auf keinerlei Weise..." in no way at all/ definitely in no way ...

Because there's a set of hats, and we're talking about things that belong to that set. "Of" is a clue, here.

It's easier to think about if we say "I don't like one of these hats." There's a bunch of hats, and "one" belongs to the bunch of hats.

I believe dieser is there because of genitive plural (adjectives get -er ending in this case).

I don't like a hat = Ich mag keinen Hut

I don't like any of these Hats = Ich mag keinen Hut dieser Hüte = Ich mag keinen dieser Hüte

Could someone please confirm I'm right or provide a correct explanation otherwise?

Masculine accusative singular -- accusative because it's the object of the verb mögen and masculine to agree with the noun Hut.

And singular because you're sort of saying "I do not like any single one of these hats."

Mir gefällt keiner dieser Hüte. is accepted now.

I don't think nirgendein is a word, and even if it were, it would have to end in -einen as Hut is masculine.

It would have to be Hut (capital H, u with umlaut) and Hüte -- but even so it sounds wrong to me. (Much as "I like no hat of these hats" and "I don't like any hat of these hats" sound wrong to me in English.)

I would say Ich mag keinen dieser Hüte and "I don't like any of these hats / I like none of these hats".

Google Translate suggests it would be "Keiner dieser Hüte gefällt mir." Is that correct?

Yes, it is. Another possibility is Mir gefällt keiner dieser Hüte.

I thought either "Keiner" or "dieser" would trigger en because it's genitive and plural, but now I'm mixed up in regards to what the subjects and objects are in this sentence.

Keiner is not genitive plural; it is masculine nominative -- the subject of gefällt.

"Not a single one ... appeals to me".

dieser Hüte is genitive: "of these hats".

keiner dieser Hüte = none of these hats; not a single one of these hats.

It's fine. "keinen" is accusative and "dieser Hüte" is genitive.