Could "antagligen" be used here instead of "troligtvis"
Yes, that sounds good, I'm adding that option now.
can I say just instead of bara?, if not, why not?
No, just in Swedish means just as in something which happened earlier in time as in ”I just saw him”. It doesn’t mean just as in only.
Is "troligen" interchangeable with "troligtvis"?
Why not "Den"?
It's a dummy pronoun, a so called formal subject, and in those cases it's always "det". Compare it to "it rains" in English. It doesn't really refer to anything, but the grammar requires it.
what about möjligtvis instead of troligtvis?
Möjligtvis = possibly, troligtvis = probably -> go with troligtvis.
Why not 'det är kanske bara min mamma'?
Kanske means maybe. Your suggestion would translate into something more like "maybe it's just my mum?" (i.e., the person is unsure of, for example, who knocked on the door or something) in English.
Vad betyder 'nog' ?
It means "probably" here.
You might encounter it other sentences with the meaning "enough" as well.
Can you get away without saying "bara" here? It feels like it should translate into, when coupled with "troligtvis", " probably only". So I would assume you could get the same message across with just "troligtvis".