Not sure I understand what you mean by "just a phrase"?
I believe it would always be more sincerely asked than the English/American "How are you?" greeting, which I have discovered is used completely regardless of the fact that the other person has no interest what-so-ever in knowing about your current state (physical or mental).
This question would certainly demand some kind of answer pertaining to events occurring yesterday, whether the person asking is actually looking for insights or is using this sentence to insinuate that the other person might have done something wrong yesterday. By the latter I mean that it could certainly be used as a "what the f... was wrong with you yesterday?" where you might be looking for more of an apology than an explanation :-)
I am a native speaker of American English. It is not correct to say that one would never use "der" or state "there" in this sentence; credit should be given for the translation "what happened to you there yesterday". If the person asking the question has reference to a specific location and not just a general question about the individual's state of being "what happened to you there yesterday" is perfectly appropriate in that situation. Both translations should be allowed as we have no context to know if the question is just casual and general in nature or site specific in its reference.