I am quite sure there's some rule for their usage but as a Czech person, I kinda just judge by my feeling (while knowing that's definitely not the best advice to give on a website as such :) When "jemu" is not the DIRECT, IMPORTANT part of the sentence, you would generally change the word order and just use "mu" (as in example "Řekni mu to" = Tell him). On the other hand, when it's the part you want to put stress on, you use the full form "jemu" (Řekni to jemu = Tell him (but noone else, just him)). For example in the given sentence, it's the interest of the asker to find out whether it was told to "Guy 1" or "Girl 2", so "jemu nebo jí?". If you just generally pointed at a person and wanted to ask if they have been told, you would ask "Řekl jsi mu to?").
Aha, now I see what you meant. Yes, because we have an object in the Czech sentence (to), we need "...tell/say it TO him or her" on the English side. Without the to in the Czech sentence, "TELL him or her" is fine, but "SAY him or her" doesn't work. But I don't know if the Czech sentence must contain to, or if it would be a valid (but different) sentence without it.
I think it should be accepted. (Did you tell him or her?)
"Řekl jsi mu?" - it means rather "Did you let him know?" or "Did you instruct him?" e.g.: "On pořád nejde. Řekl jsi mu?"
"Řekl jsi mu to?" - it means, I think "Did you tell him?" or "Did you tell it to him?"
In the reverse course, similar translations are accepted and sometimes they are even in the main translation.
e.g.: "Moje rodina by mi to řekla." -> "My family would tell me."
They can't. The word order dictates "jemu" here. With "mu" it would have to be "Řekl jsi mu to?". The question is what is the unknown new thing in the question. Are you asking whether he told him (you know whom) it or not? => "Řekl jsi mu to?" Are you asking whom did he tell that? => "Řekl jsi to jemu nebo jí?"