The difference is context as you say, but "en" can absolutely mean "one" before a common-gendered noun. The distinction isn't usually important but if it's the number you want to emphasize, you'd like on the en/ett a little more.
"Jag har bara EN grön kopp." = I only have one green cup.
"Jag har bara en GRÖN kopp." = I only have a green cup, suggesting almost that my only cup is green.