Word by word translation are not always correct. To say "I never work" in Polish you use a double negative (this is different to English!), see https://en.wikipedia.org/wiki/Double_negative#Slavic_languages.
"Ja też (nie)."
But only if this suits the grammar of the original sentence. If you for example say "Nie podoba mi się ten obraz" (I don't like this painting, or literally, To me this painting is not pleasing/attractive), 'me neither' will change into "Mi też (nie)".
I think the 'nie' part is not absolutely necessary, but sounds more natural.