In English that would be interpreted as a command without the subject pronoun. That wouldn't really make any sense, and is definitely not what the Spanish means.
You didn't see the milk would be No viste la leche. I think it is the negative that is messing you up, along with this somewhat unusual sentence. Did you see is the past tense of do you see.