I'm not a native Japanese speaker, and I'm completely guessing here, but the imperative form could be being used here in a similar fashion to the English phrase "make yourself at home". You might physically be at your place of residence, but you're not really "at home" until you take you shoes off, put away your belongings, and sit back to relax(/do chores TToTT), so the phrase お帰りなさい is basically telling someone "now that you're back, really make yourself at home."
Also, お帰りなさいませ sounds very, very formal to me, and not something one would expect to hear in normal Japanese households nowadays. Perhaps before servants would use this phrase to address their masters' return.
Yes, I agree.
I might even have been wrong because おかえりなさいませ may also be some kind of imperatve form... But I probably won't dig deep down further as that's not my genuine interest. In daily life I don't think we need to use おかえりなさいませ unless we take up a job in a restaurant or hotel, while we see and hear a lot as a guest.
Maybe it is part of the whole conversation, because the one coming back says ただいま (只今 now, I am not sure what the difference between this and 今 alone is and the dictionary did not help me). Maybe it's a "I'm returning now" - "Please return (=please do so)" kind of conversation that has been shortened to "now" - "please return". Side note: I have noticed the use of ただいま in many other situations. Like on the train (yamanote line) まもなく…です in the screen announcing the next stop becomes ただいま…です once the train arrives at the … station.
It's definitely part of a conversational ritual that Japanese people carry out everyday, which is kind of what I was getting at with my earlier comment. Technically, お帰りなさい is an imperative form, but like the English phrase "make yourself at home", we don't really think of it as a command; it's just a pleasantry that people exchange in a certain situation.
ただいま is another example of this. It literally means "just now" or "only now", and the sentiment behind it (in the case of the greeting) is along the lines of "I've just arrived home now".
As for your train observation, まもなく means "shortly" (literally "without pause") which is why it is used as a train is about to arrive at a station. As I mentioned, ただいま means "just now", so it's used once the train has already arrived.
なさい is an auxiliary verb ending which makes the verb a polite imperative, or command, though it's more direct than ください (which is a request, rather than a command) and so can sound somewhat condescending.
You add it onto the verb stem (or -ます stem/base), which means conjugating a verb to ます form then dropping the ます, e.g. 帰る (root verb) -> 帰ります (ます form) -> 帰り (verb stem).
おかえりなさい is actually something of a special case, being a greeting, so generally it's doesn't make sense to just slap the お honorific on like this. Compare:
- お帰りなさい = "Welcome back" (greeting)
- 帰りなさい = "Come home" (command)