Why not "I had to study languages"?
"I had to study languages" = "Musel jsem studovat jazyky"
Could be true, but one of the correct translations according to the system, "I was to study languages", sounds very weird to me.
Yes, that is possible in English:
He was to = he was supposed to
Fo example, "He was sent to Oxford with the understanding that he was to study languages, but he took up maths instead."
Why not "I should study languages"?
That is quite different. The original is irrealis, a retrospective thing you can't really do. Měl jsem is the past tense.
I should study is a subjunctive mood you can still manage. Měl bych se učit.