"That is a window."
Translation:Az egy ablak.
You can but it sounds very much like "the window". So, make sure you say what you mean to say.
With "Az szék" (that is a chair), it works better because of the "A"-"Az" variation.
The chair - A szék
That is a chair - Az (egy) szék.
So, even if we leave out the "egy", you can tell the difference.
With "Az ablak", there is no telling.
The window - Az ablak
That is a window - Az (egy) ablak
But in the right context, it could be easy to do so:
Ez szék, az asztal, az ablak. - this is a chair, that is a table, that is a window.