Uhhh... "Oyasumi" means vacation, rest or break from something. However, this different than when people say "Oyasuminasai." They're using Base2 command form in this situation. Yasumu = to rest. Yasumu>B2=yasumi. Add an honorific "O" and 'nasai.' O-yasumi-nasai. In other sitiuations, they would just say "oyasumi" when referring to a break from school, work, normal routine, etc. It's not the informal way of saying "oyasuminasai." If you said oyasumi to a Japanese peraon, they'd be waiting for you to talk about a rest or break of some sort.
In English, "good evening" is implied as a greeting or a farewell. "Good night" can sometimes be synonymous with "good evening" but it can also mean that you (or the person you're talking to) is headed to sleep.
Because of its dual meaning, Duolingo generally uses "good night" when referrering to sleep (oyasuminasai) and "good evening" when referring to the greeting (konbanwa).
But it's also used when you're leaving a gathering, say a party or a dinner gathering, and it's late.
It's kind of implied that you'll be going to bed when you get home,but not necessarily expected. It does signal that you're not expected to be heard from again that night.
It's more like "good bye" for tonight, if you're not at home (or you are at home saying it to guests who are leaving).
But if you're at home and saying it to family members/house mates, (or guests that are not leaving) then it means you're going to bed now.