Translation:Would you like coffee with milk? Yes.
In this context it would (well, there’s no I in bu toil, but that’s basically the idea, you repeat the verb of the question, literally: Would it be your will…? Would).
In my opinion I would should be accepted – as that’s the closest direct English equivalent of such answer, but yes is more natural translation and these sentences teach you how to answer yes/no in Gaelic.
I don’t see any reason not to accept more literal I would, I wouldn’t, etc. though – but maybe the contributors have their reasons.