Well, there is this youtube video which has a good guide on it. Here's the TL;DR version:
Mange is for subjects you can count.
Mye is for subjects you can't count.
Masse is more of a spoken form
(And Riksmålsforbundet says you should never use "masse" in written form, as it's childish even in spoken form)