That's a Duolingo-wide bug caused by the fact that the algorithm doesn't understand context.
OK, so if you write "We had done it", you can abbreviate it to "We'd done it", right? So the algorithm automatically accepts "We'd done it" even if this abbreviated form wasn't explicitly put among the accepted answers.
So here, the accepted answer is "We had to buy it"... and the algorithm not only accepts "We'd to buy it", but may even suggest it :/ And all this is because it doesn't understand that this is not a situation in which "we had" may be abbreviated.
I hope they will finally fix this...