"Musíš se víc učit." means "You have to study more". This was given as Duolingo's correct translation. "Musíš se víc naučit." means "You have to learn more." This was given as "tragram's" comment. Is this the correct difference between the two sentences?
Although the meaning in English would be the same, I wouldn't be surprised if "need to" is not accepted because the verb in the Czech sentence is "muset" and not "potřebovat." (Just my guess, based on a similar experience...)
"You've to study more" was accepted, but that sounds odd in English, we don't normally use a contraction there. Probably a duolingo thing, however, since I'm not sure that it's actually incorrect, per se.
Like you, I don't think "You've to study more" is grammatically wrong, though it definitely sounds very odd. The more usual constructions would be "You have to study more" or "You've got to study more." I'm sure it's difficult to decide what to accept, even if it's not "perfect," and what not to accept under any circumstances... so many variations!