The -o is almost always silent unless you’re enunciating a lot. This also goes for the other tens on -tio, like femtio, sextio, sjuttio etc.

Fyrtio is pronounced as if it were spelt förti. Tjugo is usually pronounced tjugi or tjuge and tju- in compounds, like tjugotvå [tjutvå]. In Finland and in older Sweden Swedish tjugo can also be tjugu and there they usually pronounce the full word in compounds as well, and don’t reduce it to tju-.

Sounds fine to my native ears. I guess it's a matter of tuning one's ears to the language.