This is what's known as the "Т-К" construction.
The word "What" is being used in two different ways here.
It's used as a nominative: "То очень грустно". But also as an accusative: "Что вы говорите".
In Russian, you can't use two cases with one word, you need one for each case that's required.
In cases like these, Russian uses constructions like "То, что", "Туда, где", "Тогда, когда", etc.
Think of it like "That, which you are saying, is very sad".