"I do not do what they want, but what I want."
Translation:אני לא עושה מה שהם רוצים, אלא מה שאני רוצָה.
Because then it would be an incomplete phrase. אבל means but as in “I like carrots but I don’t like pears”, whereas אלא means but as in “I don’t like pears, but (rather) apples”. You can see אלא as a sort of “instead”.
So here if you use אבל, you’d need to follow up the phrase with something else. Eg. “I don’t want what they want, but what I want [is not available]”.
I wrote אני לא עושה את מה שהם רוצים like you said, stricly applying the direct object rule with את haha, and it was accepted. My thought is, perhaps it is optional in a subordinate clause, whereas it would be mandatory in אני עושה את זה for example. I don't know, this is pure speculation...