"Merhaba" is more formal than "Selam". It is used when you meet someone for the first time (e.g. when your friends introduce their other friends to you, you say "Merhaba, memnun oldum."), but after you get to know them for a while, you can say "Selam!". If you say "Merhaba" to a close friend, they can react like "Um, we became strangers now?".