Why is "אתה מספר אחד עשר" not acceptable?

Because ordinal numbers are always in the female. יש פה שנים עשר אנשים ואתה מספר אחת עשרה. It's not a number of people, it's an ordinal number.

I meant, שנים עשר אנשים counts the people so is in the male form. מספר אחת עשרה is ordinal.

So after עשירי ordinal numbers don't have the yud or yud-tav in the end and sound like cardinal numbers?

If I remember correctly, ordinal and cardinals are the same for numbers 11 and up. The numbers 1-10 have different ordinal forms. Particularly the number 1. I assume the course will go into it in another lesson.