Because "in silence" sounds odd in this sentence. It might make sense with some tweaks and by setting off "in silence" from the rest of the clause (e.g. "There he remained, in silence, until the meal ended."). But without those, it just hits the ear wrong.