I have written 'μεγαφωνο' with an omikron and it was accepted, but how do I know whether to use omikron or omega? Is there a rule, or you just have to learn the words by heart?
There isn't any rule! You have to learn it by heart, except the ending of the most verbs, which is always in omega (Ω upper case, ω lower case).
For example, αγαπώ (love), τρώω (eat) e.t.c.