In general, the pronounciation at the end of any sentence should be silent (Not Osta-tha-tan, but rather Osta-tha). Also, it is uncommon (or informal) to use "kayfika" in formal Arabic, the common greeting is "Kay-fal Hal" (كَيْفَ الحَال)
Ma'am is not equivalent to "Osta-tha", as it is misguiding. "Say-ye-da-tee" (سَيِّدَتِي) for Ma'am or "Aa-ni-sa" (آنِسَة) for Miss.
"How are you, teacher?" Should be: "كَيْفَ الحَالُ يَا أُسْتَاذَة؟" "How are you, ma'am?" Should be: " كَيْفَ الحَالُ يَا سَيِّدَتِي؟"
If you have really heard "ustadha-tan" after "yaa" harf nida, grammatically it's wrong (if we follow either MSA or Classical Arabic) - it should be "ustadha-tu". I, myself, never hear the "-an" sound after harf nida from any nahwu book or explanation.
Also ... Formally speaking, pronouncing the last ending sound at the end of the sentence (according to qawaaid nahwiyya) is not a mistake.