"How do you do, sir?"
Yes, "How do you do" is closer to "Hello" in usage, than to "How are you".
The correct answer to "How do you do" is "How do you do". It is used as a formal "Hello". Personally, I would translate it to "Merhaba". Because both parties say the same thing. It is simply a greeting.
With "How are you", you can answer with the same "How are you". Or you can describe your health situation. "I'm fine, thanks". There are countless variations. So it is closer in usage to "Nasılsın(ız)".
As far as I know, "How do you do" is very formal, and very English. They don't really use it in American English.
About the origin of "How do you do": http://www.phrases.org.uk/meanings/how-do-you-do.html