"Az" can mean "that" but in Hungarian, when it's the first word in a question, it can also mean "is that?"
If you are talking about standard written English, Is that a chair? is a neutral question, while That's a chair? is what I call a "surprise/confirmation question" (I'm not sure of the proper word) -- you would use it only if you had just heard someone call it a chair and you found that surprising and wanted to confirm that you heard correctly, so you repeat the statement but with a question intonation.
How do you say:"That chair?". Context. Some asks me to go sit on the chair, there are two chairs to sit on. I point at one of the chairs and say: "Az szék? (That chair?)". Can this be correct? Or does it need to be: "Az a szék?"