You distinguish these to situations through the grammatical case of "стадіон."
- If it is the locative case (на стадіоні) then it is "at the stadium." The preposition has a static meaning if used with the locative case. Mnemonic: locative - location.
- If it is the accusative case (на стадіон) then it is "to the stadium." The preposition has a dynamic meaning if used with the accusative case.
It is a general rule and it works for other prepositions too. For example, "в дім" (accusative) is "into the house"; "в домі" (locative) is "in the house" (inside of it).