"Demek" is used with direct quotes, or quote-like phrases:
"Geliyorum," dedi. ; Hayır dedim. ; Yeter dediler.
"Söylemek" is used with reported speech:
Geldiğini söyledi. ; Hasta olduğunu söyledi.
Other than that, both "Ona dedim" and "Ona söyledim" (I told him / I said to him) are fine.
There are certain expressions that only allow one and not the other, for instance: yalan söylemek to tell a lie, (and never yalan demek).
English has these peculiarities too. You can say "to tell a story", but not "to say a story", for example. So, these instances must be learnt as fixed expressions.