Seems like a reasonable question - even if the answer is "no."
Your reasoning doesn't follow, though. In Esperanto you actually do need da if you express the number as a noun - like a dozen apples or four score years - even though you don't say "of" there in English.
- Mi volas mangxi milon da pomoj.
- I want to eat a thousand-count of apples.
Other questions in this unit which discuss "billion" require the use of "da" to be counted correct, even though the sentence is fundamentally the same as "mil" which isn't taking "da." Since as you say the noun ending -o turns the word into a unit of measurement, does that mean that the -o in "milard/o" may be excluded and thus used the same as "mil," "cent," or "dek?"