Personally I'd say the difference would be because 'in' suggests you also went into the apartment, which may not be the case. 'At' suggests you dropped off the girls rather than you spent time with them inside. Hope that clears up. Without context, either could possibly be seen as correct though.
In English, "at the apartment" could refer to a variety of different situations, but I think generally the girls would be expected to go or be inside the apartment. For instance, this sentence could refer to a situation where a Lyft driver pulls over in front of an apartment building so that his passengers can get out of the car and go inside the apartment -- he leaves them "at the apartment" by dropping them off outside of it, but they intend to go inside afterwards
It could also refer to a situation where the girls are inside the apartment when the speaker leaves. For instance, perhaps someone could be narrating a story in the present tense: "So, your sister invites me and the girls over to your apartment for brunch, but when we sit down to eat our waffles, we realize that we don't have any syrup, so I leave the girls at your apartment and run to the store to buy some [syrup, that is, not girls]."
If the girls are expected to stay outside of the apartment, I would say something else instead, like "in front of your apartment," and not "at."