You will still be understood if you remove it, but you will sound a lot more natural with it. It can be compared to the English "hey" or "o" as in "o Bob", which might sound formal in English, but it's very often used in Arabic when speaking directly to someone. You need to listen to a lot of Arabic every day to have a feel of when to use it, but the good news is that it is used so often, that you'll get used to it earlier than you'd expect, then you'll start wondering how you've ever lived without it :-)