"He speaks perfect Swedish."
Translation:Han talar perfekt svenska.
It's an unusual word, and much more expressive than perfect, but sure, we can add it.
Why can't this be translated into "he speaks perfectly Swedish"? I thought perfekt could be an adverb as well...
It can be, but it isn't because of the word order.
The adverb would go after the object, like in English.
"He speaks Swedish perfectly" -> "Han talar svenska perfekt"
"He does it perfectly" -> "Han gör det perfekt"