No, they can't. Also it's oversimplifying to say that one's for positive and the other's for negative.
Basically, 'jej' is either genitive or dative, while 'ją' is accusative. It is very common that a verb takes accusative in a positive sentence, but genitive when it is negated. And so it the case with this exact example. So what you wrote works here and will work in many cases, but it's only 'common' and not 'always'.
No, that's wrong.
"Ten kot nie ją lubi" would be kinda correct but weird anyway (This cat likes not her, but someone else). As it negates 'ją' and not 'lubi', the grammatical case changes from Genitive in the original, negated sentence, to Accusative in 'my sentence'.
I see a lot of people creating sentences which tend to mean "not me/him/them, but someone else" as a result... seems that it's really better to keep to the rules first, and later learn how to break them.
"this" and "that" have translations dependent on the grammatical gender, not to mention the cases.
In Nominative, as here it's just the subject of the sentence so it's Nominative, they are like this: ten (masculine), ta (feminine), to (neuter), ci (masculine personal plural), te (not masculine-personal plural).
With 'that', apart from femine Accusative (tę / tamtą), you just add 'tam' at the beginning of the determiner.
And technically, "ten" is not a translation of "that". But it's an accepted solution. Polish and English look differently at the notion of 'closeness'. English has "this/that/that", Polish has "ten/ten/tamten". So the first 'that' and the second 'ten' overlap.
Well... not. The subject of the sentence is "ten kot". The verb "lubić" needs Accusative: Ten kot ją lubi / lubi ją (for a positive sentence). But every time when a sentence with Accusative is negated, Accusative changes into Genitive. So: The kot jej nie lubi / nie lubi jej.
In my first example: „lubić” is negated, hence ją->jes.
In second and third example I negated „cat” and „her” but not „lubić”. In this case „ją” stays „ją”. There is also another possibility, not shown above, that „jej” is used as a possessive pronoun. In that case jej stays jej.
A sentence like that could look like this:
Ten kot lubi nie jej mleko. (It's not her milk that this cat likes)
However this sounds rather strange.