This exact discussion was had (for years, with a little but of bitterness) amongst the Esperanto community half a century ago. The result was that the tense of the Esperanto 'ser/estar' equivalent (esti) defines the base time and the tense on the participle defines the state at the base time. Quite a few languages are odd about past participles, and all of them handle time relations a bit differently.
Leads to confusion when translating.
"have never gone" = continues into the present. At this moment, you still haven't done the action. "had never gone" = in the past, you never did the action UNTIL a certain moment where you did. Example: "I have never been to that store" (still never have) vs "I had never been to that store until this past Saturday" (now you have been)
Generally we only use the past perfect in reference to other past events. You never really use it outside of context. Think of it as the past of the past.
Aren't "I'd never been there" and "I hadn't ever been there" pretty much the same? If so, both should be accepted.
Not ever = never
Perhaps there's a different expression for "not ever" in Portuguese? If so, I haven't found it. As far as I know, it translates to nunca. Does anyone else know?