"How do you say this in Japanese?"
と is the marker for what is/was said or what something is called.
I am (called) Tom. トムと言います。
He said yes. （かれは）はいと言いました。
What did he say? （かれは）何と言いましたか？
This particle is also indicate something that was thought.
I thought it was expensive. たかいとおもいました。
I think that I'll go to the store. みせに行こうとおもいます。
「何と言いますか」('How do you say?') works as a set. Even when only 「何と？！」 is uttered, it is usually implied that the full expression is 「何と言っていますか？」 or something similar.
In addition, there is another meaning of なんと, which means 'how !'/ 'What a ___!' Placing なんと before 日本語, while it doesn't sound that natural, it sounds like you are saying 'How Japanese you say it!?'.
my answer was wrong, however the 'correct answer' at the bottom of the exercise page does not match the "translation" at the top of this discussion page! The particles are used differently, for example: In the red box at the bottom of the exercise page, the 'correct' answer begins with これを however the 'translation' at the top of this Discussion page shows it commences with これは. No consistency even within a single exercise!
It is incorrect.
何で is usually pronounced なんで and means 'why?'.
日本語といいます would be 'It is called Japanese'. 日本語で is the proper phrasing ('in Japanese').
The Japanese sentence shown as a correct answer is more accurately translated as 'What do you call this in Japanese?', rather than 'How do you say this in Japanese?'.