Mea here is an adjective meaning 'my'.
It has to match the grammatical gender, case, and number of the noun it modifies. So here, it takes the nominative feminine singular form since it is used with magistra (feminine noun, nominative case, singular). The nominative forms of this adjective are meus (masculine), mea (feminine), meum (neuter).
Mihi is the dative form of the personal pronoun ego. It's meaning in this form is similar to "to/for me".
If you used mihi here: Magistra mihi studet, it would mean more like "my teacher studies me", which doesn't sound very appropriate.