Great question! ;D
Well, literally dementor either means "one who causes dementia", or it comes from the original Latin word dementia or demens, which more or less means "deprived of mind/spirit/soul", in which case I guess dementor would mean something like "one who takes away minds/spirits/souls"
Dementia is demenco in Esperanto, so I guess demenciganto for the first possibility, and for the second maybe something like menseliganto. That's from menso (mind) + eligi (to take out, remove; from el + igi) + -anto
Or perhaps with -ulo instead of -anto? (demencigulo / menseligulo)
RaizinM is correct about the etymology. But it's not necessary to capture the entire meaning in a compound word. Readers would come to understand what the word means in the same way as readers do in English: from examples and experience.
Demensulo and formensulo strike me as good candidates, with the second being more precise but the first preserving more audible similarity to "demenc/o".