In the context of Orthodox Judaism, it makes sense. "Not touching" means literally not touching people of the opposite sex.

מה? בנשים?

The inflected form "(do) touch" is נוגע; The infinitive form "to touch" is לגעת.

I was absolutely sure you were a native speaker O:

It does sound a bit weird, unless inferred from the context.

In Orthodox Judaism, it refers to not touching a woman

Hmm... not really. "I am not relevant" would be something like: "אני לא קשור (לזה)" or "אני לא נוגע לעניין" (which sound a bit awkward) or (shifting the subject): "הדבר לא נוגע לי"