"Sie" has different meanings: she, you (formal) and they.
You can tell them apart by:
1) the verb. "Sie" in the sense of "she" is used with a singular verb (3rd pers. sgl.), and Sie/sie in the sense of "you" and "they" is used with a plural verb (3rd pers. plural).
2) the capitalization. When "Sie" means "you", it is always written with a capital "S". When "sie" means "she" or "they", it is only capitalized when it is the first word in a sentence.
Mag sie das nicht? = Does she not like that?
Mögen sie das nicht? = Do they not like that?
Mögen Sie das nicht? = Do you not like that?
When someone says "Mögen Sie" to me, how do I know the "Sie" from "sie"?
Personal pronouns refer back to something you have already talked about.
Have you just been talking about some other people? Then the /zi:/ you hear is probably sie "they".
Have you not been talking about anyone else? Does the person look like they're interested in your opinion? Then the /zi:/ you hear is probably Sie "you".
Is there some subtle pronunciation difference?
No. The pronunciation is completely identical.
You don't translate literally. You translate to what a native speaker would normally say.
I wrote "Do they dislike that" (I have always used dislike as a translation for "moegen ... nicht" as it sounds most normal to me) and it was marked incorrect. It has been accepted in other circumstances just not here. That leads to my other question, is "moegen ... nicht" the only way to say dislike or is there a better way to say it?
A literal word by word translation would read as "Like they that not" Getting the correct word order here is mostly guesswork based on the question mark, Earlier DUO suggested that NICHT usually means Never when used in this way so "Like They That NEVER?" which is even more cryptic I have now decided to ignore the "NEVER" advice given by DUO previously.
I am an English speaker, I found this confusing, purely because of the word order. I instinctively thought that the answer is "They do not like that" I know that there are rules about word order etc the only clue that it was the question was the question mark - Do they not like that was the result when learning any new language most people, myself included, tend to concentrate on learning new word translations the subtleties of German Grammar are simply far too exasperating Mark Twain was correct with his essay regarding learning the German language - https://www.cs.utah.edu/~gback/awfgrmlg.html
Why is "don't they like that" wrong?
It's not. And it's one of the accepted translations.
Without a screenshot showing exactly what happened, I can't say much more.
The most common cases do involve some mistake on the learner's part, though, rather than a problem of Duolingo's -- overlooking a small typo, or mistaking a listening exercise for a translation exercise.