"The guests are sitting around the table, under which a big dog rests."
Translation:A vendégek a körül az asztal körül ülnek, amelyik alatt egy nagy kutya pihen.
The word "ami" would refer back to the whole first clause, instead of just the noun "table". That's why. Here, "amely" or "amelyik" is the correct usage.
Here's how you could use "ami":
"The guests are sitting around that table, which is very nice of them."
"A vendégek a körül az asztal körül ülnek, ami nagyon szép tőlük."
But the Hungarian usage would probably prefer breaking this usage into two sentences:
"A vendégek a körül az asztal körül ülnek. Ez nagyon szép tőlük."
There is another usage of "ami", when it refers to a noun that is not named. The noun here is "table". It is named, mentioned in the sentence. Consequently, "ami" can not be used. Here is how it would be OK to use it:
"The guests are sitting around under what a big dog rests."
"A vendégek a körül ülnek, ami alatt egy nagy kutya pihen."
See, the noun ("table") is not named in the sentence. It is just implied that they are sitting around something. So that is when the usage of "ami" is actually correct.
Here is an extended discussion of the topic: