El- is a bit a weird prefix. :)
Its main role is giving the movement a perfective aspect. The short rule is, if you give a target, it means "to go to the target", and if you don't, it's better to translate with "to go away". What matters is that you're in a different place at the end.