"Hi, what's your name?"

«Привет» is an informal greeting, and «вас» is a formal pronoun, so it’s not a good idea to use these two forms together (unless you’re using «вас» as a plural — it can be used in informal speech, but then the English would be "What're your names?").

Well, it depends. In some situations, e.g. on an informal party or in the university when talking to fellow students, it might be acceptable. At my work, we have a guidelines of using «ты» to everyone, I was explicitly told this when I started this work (although I keep using «Вы» to most people at work anyway ^^').

In general, I'd recomment to use Вы whenever you're unsure if ты is acceptable.

Also, if someone addresses you with ты, or uses привет or other informal words, you'd also use ты when answering them in most cases.

«Как зовут тебя» marks «тебя» as the most important information in the sentence. You could use it if you're asking several people's name:

— Как тебя зовут? 'What's your name?'
— Иван. 'Ivan.'
(To a different person:)
— А как зовут тебя? 'And what's your name?'

Here, 'naming' is established context, previous sentences were about how people are called, so you give emphasis to «тебя» because it's the part of the question that is different from the previous question (even though the previous sentence also had «тебя» as an object, it referred to a different person so we emphasise «тебя» to show it's the new piece of information).

But in most cases «Как тебя зовут?» is the only word order that sounds natural.

«Как тебя зовут?» is not translated directly, the Russian sentence is closer to ‘How are you called?’.

Yes, «у тебя» is ungrammatical here.

«У тебя» is ‘at your possession/at your place’, and «тебя» is ‘you’ in the accusative case, used for direct objects (literally the Russian sentence means ‘how do [they] call you?’).

The usage of «вы»/«ты» is really complex, and ‘don’t use «ты» to people you don’t know’ is just a guideline. If you follow that guideline, you definitely won’t be rude. But this is not cut in stone.

In some situations you can use «ты» to strangers, e.g. to a fellow students (or in other situations where you’re both quite young and you’re peers) or when speaking to children.

When I just got my work (I’m a programmer), my boss asked me to use ты to everyone in the office.

The usage of ты/Вы is really variable, it depends on the place and even on social class. When my Grandpa’s sister from the Russian Far East came to visit us, she used «ты» to shop assistants. This came off as very rude to us, but she said she didn’t intend to be rude, she just used «ты» because those shop assistants were obviously much younger than she is. For her, the age was an important factor.