"Why don't you like this book?"
Translation:¿Por qué no te gusta este libro?
Este is a masculine singular adjective that can be used before a m.s. noun: Me gusta este libro. (I like this book.) It can also be a pronoun standing for or replacing a m.s. noun: Me gusta este. (I like this one -- meaning book.)
Esto is a neuter pronoun. It can't go before a noun. It stands for a situation, occurrence, or circumstance (Esto es sorprendente. This is surprising.) or for something you can't tell the gender of (¿Qué es esto? What is this?)