A native Swede can come in and correct me if I am wrong, but if a word has an adjective and an adverb form, the adverb form is usually the neuter form (there are some words that have their own adverb form and some that exist only as an adverb). IE "en långsam man" is "a slow man" but "en man springer långsamt" is "a man runs slowly." (I adjusted it so they both have en man)
A general rule of thumb that has helped me is that when not using a noun, Swedes tend to default to the neuter form. For example the sentence "it is raining" is "det regnar" or "it is cold" is "det är kallt." Notice that the "it" in these sentences are "det" and in the latter I used kallt instead of kall. As far as I can tell, this extends to adverbs as well.