As a general rule, nicht + ein = kein.

I did a translation earlier which was marked wrong: "Das ist keinen Apfel", Duolingo said it should be "Das ist kein Apfel" and in the comments section I understood the reasoning to be because Das is neutral and that takes precedence over Apfel which is male.

No; the reasoning is that you need the nominative case (e.g. der Apfel, kein Apfel) and not the accusative case (e.g. den Apfel, keinen Apfel) after sein "to be".

Here, Katze is feminine and so the nominative and accusative look the same; still, keine Katze is in the nominative here, not the accusative.

Why keine and not kein?

Because Katze is feminine, not masculine or neuter.

Bcs katze is feminine and "kein" is used before neutral and masculine nouns. So the katze is feminine and the only suitable word is "keine"

Das ist der Katze nicht - no good.

Right. Firstly, Katze is feminine; secondly, the English sentence has "a cat" and not "the cat".

So you would need not die Katze but eine Katze.

But since you're negating it, Das ist nicht eine Katze turns into Das ist keine Katze.

nicht as a separate word would be appropriate for a definite noun such as die Katze, but not for an indefinite noun such as eine Katze. (A bit simplified but a good rule of thumb.)

I suppose you could say that this is negating the object rather than the verb but that's the way it would be expressed in idiomatic German.

Why isnt 'dies' correct rather than 'das' at the start

Because the English sentence has "that" and not "this".

dies is for things that are close to you = this, these

das is mostly for things that are far from you = that, those. It can also be used for close things = this, these.

das ist nicht Katze would be like "that is not cat". It doesn't work in either language, because Katze/"cat" is a countable singular noun.

What does work is "not a cat".

That might seem to be nicht eine Katze, but usually, nicht + eine turns into keine.

So you end up with Das ist keine Katze.

kein is for neuter or masculine nouns but Katze is feminine, so the two don't fit together.

It should probably have corrected you to keine Katze rather than kein Kater as that would be less confusing, but it seems to like to change the second word rather than the first if both changes are possible.

der Kater is a male cat, a tomcat. die Katze can be a female cat or any cat (regardless of whether the cat is male or female).