Translation:I like dogs, but my sister likes cats.
So presumably 'aber' is NOT a subordinating conjunction? Otherwise the sentence would end "...aber meine Schwester Katzen mag."?
That's right. aber is a coordinating conjunction. Here it's used to combine two main clauses.
It could also be used to combine two subordinate clauses:
„Er weiß, dass ich Hunde [mag], aber meine Schwester Katzen mag.“
Verb does not change its position in some relative cluses (clauses that begin with aber, und, denn...).
Could someone explain please, why 'Schwester mag' and not 'Schwester magt'?
What's the role of the coma here? Is it mandatory, only recommanded or optional?
IMO it's mandatory here. While the comma before some coordinating conjunctions is omitted…
„Ich gehe zuerst und du folgst mir.“ – „I go first and you follow me.“
…when the conjunction implies a contrast or difference such as aber, doch, jedoch or sondern, there has to be a comma, even for coordinating conjunctions:
„Ich gehe, aber du bleibst here.“ – „I go but you stay here.“
die Katze (plural die Katzen) is grammatically feminine word for the biological family of animals and it can both mean a female and a male cat.
der Kater (plural die Kater) is grammatically masculine and means specifically a male cat.
If you explicitly want to talk about a female cat, there is the grammatically feminine word die Kätzin (pl. die Kätzinnen), but IMHO it's not used that often. AFAIK the usage of adjectives is more common here:
„eine weibliche Katze“ – “a female cat”