I think there's a difference between "be in love with somebody" and "love somebody".
If I love a singer, I probably enjoy listening to their music, but I don't have to have any romantic thoughts about them.
If I'm in love with someone (a singer or whoever), then I have romantic thoughts about them.
It would be the same with, for example, "Mnoho žen mě milovalo." or "Sedm žen mě milovalo." The amount + noun form a single unit.
It is, however, possible to break up the unit - but only if you want to stress "žen" by placing it at the end or the beginning:
- Kolik tě milovalo žen? - How many women (not men) loved you?
- Žen mě milovaly tisíce. - As for women, thousands of them loved me.