"This cat is small."
Translation:Ten kot jest mały.
"kot" is a masculine noun, the right form of "this" for it is also masculine "ten". "to" is neuter.
Due to another meaning of "to", your sentence makes some sense, but it means "It's the cat that is the small one" (you see three animals, you're asked 'one of them is small, which one?' and you answer that it's the cat).