The tips and notes for Numbers 3 begin like this:
"Recall that Czech cardinal numerals from one to four behave as if they were adjectives. The counted object appears in the case appropriate for the context, and the numeral matches that case."
That is the exact opposite of "fixed no matter what", we say that "appears in the case appropriate for the context". Such appropriate case here is the genitive case and so we have the genitive "dvou".
For 41 we would either have "Ptáme se čtyřiceti jedna matek." or "Ptáme se čtyřiceti jedné matky."