bawić się is related to enjoying and not playing. In polish you play something (grać) when there are rules to follow (gram w szachy) and having fun or enjoying when rules dont apply (bawię się z psem) I think it would work also to have the translation as 'they do not want to have fun with you'
"Bawić się" is very rarely used in the sense of having fun though. Aside from "dobrze się bawić" i "dobra zabawa" (neither of which is used without "dobry" or "dobrze") I wouln't translate "bawić się" as "having fun" in any other context. Seriously, you wouldn't say "Oni nie chcą bawić się z tobą." about a group of adults.
Just so i'm sure, "bawić się" is the type of playing that encompasses "playing with a baby", "playing with the dog", or "just playing around", right? not like playing a game?
"Bawić się" is used in situations mentioned by you and when you mean kind of children playing. But according to these dictionaries "play around" can be translated otherwise: https://www.diki.pl/slownik-angielskiego?q=play+around, http://pl.bab.la/slownik/angielski-polski/play-around
For playing a game you should rather use "grać".