"Vous veniez me voir."

[cIlPUJRN]

How do you know it means "used to come"? Why can't it just as easily mean "You came to see me."?

[Prenom.Pierre]

The imperfect (veniez) gives, here, the idea that the person used to come. This is not an isolated case (You came... vous êtes venus (une fois) me voir).
The person has come many times.

[W-Ruggles-Wolfe]

Indeed. L’imparfait suggests a duration of time. (He was communicating. vs. He called you.).

https://www.cliffsnotes.com/study-guides/french/french-ii/french-ii-past-tenses/the-imparfait
https://francais.lingolia.com/en/grammar/tenses/imparfait

[PatruAloeil]

"You came to see me" and "You used to come to see me" are both accurate translations for this sentence.