Hmm, let me shape a theory here. :)
It probably does sound less awkward in speech because while speaking you do not make a difference between "use to" and "used to", so it sounds familiar. But while writing, you are bound to use "use to", and this phrase does not make much sense.
Yes, I agree. In fact in "used to" the s is unvoiced -- like the noun use, but unlike the verb use or used in other contexts. In "used to" it's hard to distinguish the unvoiced s from a voiced s, and both sound okay. But in "didn't use to" the unvoiced s sticks out, and in writing the... unusual... noun/verb conflation is hard to ignore.
Edit: Well hmm I agreed, and then went on to make a distinction between some pronunciations of "use to" and "used to." So let me clarify that normally the s is unvoiced in both of them, and they do sound very similar; but you can voice the s in "used to" without it being too jarring, whereas you cannot in "use to." Or at least I can't.
My dialect is Midwestern US, and I've spent a lot of time in the East and West of the country as well. I don't think I've heard any native speaker voice the s, except when speaking slowly and with emphasis -- but it's quite possible that I just haven't been paying enough attention.
Here are a couple on Forvo, which both sound unvoiced to me (one from Australia):
Now I'm curious: in what dialect(s) do you hear the s voiced? How would you pronounce "I'm not used to drinking orange juice, though I used to drink it a lot"? Me: yoosta, ornch, yoostoo. I know that not everyone pronounces "orange juice" this way.