Why isn't "ik moet niet zwemmen" correct?
Yes, but that suggests that you should or must not swim instead of simply not needing to.
Why is it not possible to say: "Ik hoef niet OM te zwemmen" ?
"Hoeven" can also mean "wanting." But pretty much only in negations, it will be "not necessarily wanting to" though, or "needing."
"Ik hoef niet te zwemmen" would then be "I don't necessarily want to swim" or "I don't need to swim."
so for most auxilary verbs the te isnt needed, but for hoeven it is?
Exactly: hoeven always needs te before an infinitive. :)
...while the other modals doesn't? (Optional or not?)
Yes, the others never take te. :)
Why can not be " Ik heb niet te zwemmen"?
This is not translated with hebben, but with moeten or hoeven (when negated). Since this sentence is a negation it's Ik hoef niet.