I am not an American, so I rather resent the US flag representing my language!
The main problem the way I see it, is that flags represent countries, not languages. Ideally, flags shouldn't be used as symbols for languages at all. The issue then is, of course, that few languages (I can only think of Esperanto) have symbols representing them. So the alternative is often just the name of the language in text, which is not very effective from a visual point of view.
Regarding Swedish, this is a problem because Swedish is spoken in both Sweden and Finland, but you only ever see Sweden's flag representing the language. Now, for a Swede, that's not a huge deal, because we wouldn't think much about it. Sadly enough, we're used to the concept of Swedish=Sweden. Consider on the other hand a Swedish-speaking Finn. Born and raised in Finland, with Swedish as their native language. But everywhere they go, their language is represented by the flag of a foreign country, and the flag of their own country is used for the other official language. The effects of such usage is that the common misconception of Swedish being a foreign element in Finland, is strengthened. The same could basically be said about Finnish speakers in Sweden.
English analogies could be the Scottish flag being used for Gaelic, while English being represented by the flag of England. Or the Canadian flag representing French while English being represented by the US flag (I've actually seen this happen on products, it honestly amazed me).
I ended up ranting now, but I do think that this is a problem, because this usage reinforces the idea that languages belong to a single particular state, and in that state there is one "correct" language. Unfortunately I don't really know what could be done, since there really is no alternative that is as visually effective as a flag.
Duo teaches American English, although Commonwealth spellings of words are also accepted, so it makes sense for them to use the American flag. In addition, for ease of recognition and simplicity of coding each language has just one icon, so the Spain flag has similarly been picked for Spanish and the Brazilian flag for Portuguese.