"When did you people go to Delhi?"
Translation:तुम लोग दिल्ली कब गये?
Only transitive verbs (verbs which can have direct objects) agree with their objects in the past tense. Intransitive verbs (verbs which cannot have direct objects) agree with the subject in all tenses.
'To go' is an intransitive verb because you can't say 'go Delhi'. You need to say 'go to Delhi'.
Even in Hindi, जाना is intransitive. Though it appears like दिल्ली is a direct object of गये in this sentence, there is an implicit को between them that's dropped - तुम लोग दिल्ली (को) कब गये?
Therefore, the verb जाना always agrees with the subject, which in this case is तुम लोग.