are there any set rules that would simplify knowing which of the greek letters should be used for the English "O" and the English "S"?
For English "S", it's easy: use ς at the ends of words and σ everywhere else (beginning or middle).
For English "O", there's no way to know whether ο or ω is appropriate in Greek, unless you know whether the word used to be pronounced with a long or a short vowel 2500 years ago -- it's like English "ee" versus "ea", which used to be pronounced differently and were therefore written differently but are now pronounced the same and you just have to learn that it's "treat" but "tweet" etc.
Same story with η ι υ οι ει.
One small ray of hope: the above is true for word roots, but endings are more predictable. For example, the noun ending "-os" is -ος while the adverb ending "-os" is -ως; the passive ending "-ete" is -εται while the "you all" ending in the indicative active is -ετε; and so on.
I am confused about O and H (English is not my navite). Why O means ,,the" but not ,,a"
ο, η, το are the Greek definite article, corresponding to English "the".
Roughly speaking, the definite article in both languages is used when you are speaking about something that is already known to the listener - often because you have spoken about it before.
The indefinite article ("a, an" in English) is used (roughly speaking) when you are speaking about something new.
For example: "Yesterday, I saw a man. The man was selling balloons."
In the first sentence, he is "a man" because he is new information; in the second, he is "the man" because the listener already knows which man you are talking about.
The use of definite and indefinite articles is a bit more complicated than that, and not quite the same in English as in Greek.