That is the only translation possible. It's just a bit of a strange sentence.
I really dislike this kind of strange sentence, because I tend to dismiss the actual translation as making little sense which then leaves me scratching my head as to what the sentence is supposed to mean.
You say it's the only translation possible, which may well be true, but I see two possible interpretations. Is it someone taking milk to a cafe or someone going to a cafe that already has milk? The English sentence is ambiguous. Is the Welsh sentence similarly ambiguous?
Yes that is one interpretation of the sentence in both English and Welsh, but to be more specific about taking something somewhere you say "Dw i'n mynd â llaeth i'r caffi".