Szépen literally means 'nicely' (szép is nice, pretty), and nagyon is 'very'. You can both use them with köszönöm, or even together, but for reasons barely anyone knows about, they have a pretty fixed structure in this idiom:
Köszönöm szépen; nagyon köszönöm; or nagyon szépen köszönöm if you want to be super polite.
Thank-you for sharing your expertise Ryagon! It is much needed here. You are helping others learn your beautiful language, and you are opening Hungarian to the world through your knowledge of English. Please keep posting. It would be hard for us foreigners to know about word order in idioms like this without living in Hungary.
Sometimes to make fun of this super polite tendency, I use nagyon szépen köszönöm szépen which is a compound of nagyon szépen köszönöm és köszönöm szépen. Those whom I say this to usually find it really funny and especially elderly people welcome it gladly. (For a foreign speaker it may be taken as a funny mistake and in certain circumstances it may sound childish, so mind twice to use it—but it may give you nice moments if you crack it in the right time. ;) )
°"Köszönöm nagyon" is wrong, but "Nagyon köszönöm" is quite often used. We use the... (sorry, I am not familiar in linguistic terms) qualifiers(?) in front of the qualified word, like "gyorsan megy" --> goes fast (lit. fastly goes), "szépen repül" --> flies nice (lit. nicely flies), "nagyon elfoglalt" --> very busy.
"Adverbs", in this case. :)
Descriptive adjectives and adverbs are in front of the described words. Köszönöm szépen is an exception to this, but fixed phrases get some more grammatical wiggle room. Szépen köszönöm thus isn't inherently wrong but might sound a little off, maybe even a little rude for presumably leaving the nagyon out in front.
It should be since technically all the ways of thanking are synonymous. But köszönöm szépen is much more polite and elaborate than your ordinary 'thanks' or even 'thank you'. To get a feeling for it, I'd suggest the following translations:
- Kösz / Köszi - Thanks
- Köszönöm - Thank you
- Nagyon köszönöm - Thank you very much
- Köszönöm szépen - Thank you kindly
- Nagyon szépen köszönöm - Whoa, now really. I'm amazed. Thank you.
Because it is not one of them. Hungarian is a Finno-Ugric language, a distant relative of Finnish and Estonian ("distant" means like English and Persian in this case). Hungarian and Finnish are not mutually understandable, the similarity is not even noticable if you don't know what to seek.