- Igen, ez a szék.
When you use determiners ez/az (this/that), you ALWAYS have to use the definite article, even though it doesn't translate to anything in English.
That means "that window" would therefore be "Az az ablak"
You also have to decline it based on case, but don't worry about that until way later. If you're curious though, if "this window" is a direct (accusative) object, you would have "ezt az ablakot", and "on that table would be "azon az asztalon."