"Is that a chair?"
You don't need to use a form of "to be" in this case- van when a sentence is merely stating what or how something is.
So "This is a chair" is just "Ez (egy) szék," and "This chair is tall" would be "Ez a szék magas."
You do however use van when the sentence has an adverb or a question, such as "Hol van a szék?" - "A szék itt van."