Too complicated to explain in brief, so check out this fairly detailed page.
In short, it just a way to modify a noun that entails the action being remembered, recalled being experienced.
Here a copy paste about a relevant section:
>In situations where the object is being described by some action that another person did, the subtleties of ~던 seem more apparent. For example, I could say these sentences:
>이 빵은 슬기가 자주 먹던 빵이야 = This bread is bread that Seulgi eats often
>이곳에 근무하던 우체국 직원이 삶을 영원히 포기했어요 = The worker/employee who used to work at this post-office gave up on his life forever
>길을 밝히던 불이 꺼지고 나서 길이 어두워졌어요 = The street became dark after the light that used to brighten the street went out
>방금 들어온 사람은 우리 가게에 자주 오던 손님이야 = The person who just came in is a customer who comes to the store often
>I use the present tense “come” in the English translation above. You could argue than “came” or “used to come” would be more appropriate because we are describing the person as a customer who came in the past. However, because of the context of the sentence – they also “just came in” and are probably still there now – indicating that it could be used in the present tense. Again, it is difficult to translate these sentences in English. Pay more attention to the Korean sentence and use the English translation as a reference.
>Because ~던 (through the usage of <sub>더</sub>) indicates that one is “remembering/recalling” something, the bread is not only “bread that Seulgi eats often” but more specifically “bread that I specifically remember/recall/experienced Seulgi eating often.”