"It's very quiet on that mountain."
'山中' (in Japanese '山の中') has been a well established Chinese expression before Japan systemically loaned Chinese characters. That is how it becomes a Japanese word.
For an English translation, I think it is OK to say 'on the mountain'. 'In the mountain' might be a better translation but I am not sure if it is accepted by native speakers.
It refers to a space within the mountains: you go in the mountains, and there are tall trees around blocking the sunlight. You are isolated from the other parts of the world and blend into the mountain landscapes.
In the 5th century (or 4th, 6th? my memory on this is somehow blur), an emperor asked one of his minister 'what is there in the mountains'. The answer was a famous poem (please forgive me in that my English is not so proficient to make it rhyme in English)
What is there in the mountains?
Over the ridges, there grow white clouds.
Yet I can only enjoy them myself,
I cannot bring them here and present to Your Majesty.
In short, you just share the calmness you felt when you go out of the mountain.
Thanks for the interesting reply.
山中（さんちゅう）looks to be used still in that way in Japanese too.
Jisho.org has the English definition of 山中 as "in the mountains, among the mountains".
The digital version of the 大辞泉 dictionary says it means 山間（やまあい）, which Jisho.org says means "valley, ravine, glen, gorge".
So this tallies well with what you have said.
However, I'm pretty sure "in the mountain" would be incorrect for the English translation.
"In the mountains" is fine. The meaning is "within the mountains".
"On the mountains" has a different meaning to "in the mountains"
("my house is in the mountains" is correct; "my house is on the mountains" is wrong as it can't really be situated on multiple different mountains.)
"On the mountain" (singular) works, but just looks strange to be the translation of 山の中.
("my house is on the mountain" means you have a house built on the surface of the side of the mountain.)
"In the mountain" (singular) pretty much only means "inside the mountain", as in underground kind of thing.
(my house is in the mountain" would mean it's like a cave or something.)
Same with hills: "I'm standing on the hill"; "I'm standing in the hills"; but not "I'm standing in the hill."
I was sat with a Japanese native when this question came up, I put in the の中 and asked him whether this was correct as I had seen it in a previous question however from my learning I didn't think this would be correct. His English isn't great however he was confused by including the の中. Jungerstein gave a great description but as they estimated it doesn't seem to fit with how native speakers would say this.