"This book is long."
You can use both, but Duo is silently forcing you to use a specific interpretation without any real indication in the sentence other than some fuzzy, subjective reasoning. They probably want to say that since the word "this" is used, that the subject has already been established and you would then carry on with the topic marker. But no, you can easily introduce "this book" as the new subject. So, as usual, report it.
これ can be standalone as a subject. It is only to appoint an object thats why it only can be used as これは…てす means It is a ... This is a.... That is a... so on
but この can't stand as subject by itself. It is always attached to the specific object. thats why it used as
この本は…てす This book is... That (object) is/are...
を marks the direct object of a verb; the thing a verb is acting on.
There is no verb acting in this sentence though, only an adjective describing a state of being.
"Book" is the subject/topic described as "long". A person cannot do the action of "is long" to something else.
It helps to remember that の is a genitive particle used to link two nouns together.
For example it is used to show possession 私の名前 "My name" 彼の家 "His house", etc.
これ is a pronoun, "this" which replaces other nouns, (and sounds a bit like 彼・かれ the pronoun "He/him"
この is a pre-noun adjective and a contraction of これの; since this ends in the linking particle の it means it must be followed by a noun この本 "This book"
長い, as i understand, can be use to describe something as tall, wide, long... in this case, it describe the book is long as in pages. It can be pretty vague to use this sentence. It would be more accurate for the speaker to say "the page number of this book is too much" or "the height of this book is too long"