If you wanted to use "estas", you would need to say, ""Ŝiaj studentoj estas tre malsimilaj". You cannot say, "estas malsimili" - that would mean "are to be different", and doesn't make sense, either in Esperanto or in English. I hope that helps.
(Edited to add: I've just thought of a way that "are to be different" can make sense in English, but the meaning would be quite different to what we have here. "Her students are to be very different" would indicate that she, or someone else, wants them to make themselves different from one another. If that sounds rather complex, think of a sentence like, "I am to be at the station by 8 o'clock." In other words, that is the time I have been told I need to be there, to catch a train, meet a friend, or whatever.)
When I first learned Esperanto, back in the 1970s, I would have thought the same. But it seems that between then and now, the idea has become acceptable of using a verb instead of estas plus an adjective. So you can say, "Mi estas alta" or "Mi altas" for "I am tall", and "Ŝiaj studentoj tre malsimilas." or "Ŝiaj studentoj estas tre malsimilaj. " for "Her students are very different".
I have a problem with "malsimili" and "malsami". Which one describes the greater difference? Probably "malsimili" means to be so different, that the described items are even not similar in any way, whereas "malsami" means the described items are only a little bit unsimilar, so that they are not the same, but still similar. Is this correct?
..or for Brits, "Lernejano" is for school pupils (primary and secondary schools), and "studento" is for students at college or university (higher or tertiary education). I am unsure why this should be amusing, though!