Why "mutta" instead of "vaan" is not acceptable?
"Vaan" is used to introduce information that contradicts information that was previously negated within the sentence.
what the difference with "mutta"?
The meaning of "vaan" is closer to "but rather". The meaning of "mutta" is just "but".
"Mutta" is a conjunction that starts a subordinate clause, there's always a comma before it (or after the clause if the subordinate clause starts the sentence).
This might help you with the difference.