When the auxiliary verb is avoir the past participle matches the number and gender of the object not the subject, only if the object precedes the verb.
It's different when the auxiliary verb is être. In that case the past participle does match the subject.
Elle est retournée en France.
Elle a retourné la lettre.
Because the object "la lettre" comes after the verb. Here the auxiliary verb is avoir so the object could matter, but it would only do so if it comes before the verb. For example: La chemise qu'il a mise est verte [the shirt that he put on is green] Here the object que (which represents "la chemise") comes before the verb so the verb "mise" must agree with "la chemise". That is why you cannot say 'mis' here. It absolutely has to be "mise"
Just to clarify.
In the sentence in this exercise the past participle is not agreeing with "livre", even though "livre" is the direct object. The participle is unaffected because the direct object is coming AFTER the verb.
SquirlRat noted, when the auxiliary is AVOIR, the participle agrees only if the direct object precedes the verb. That is why, "retourné" is not influenced by the gender of "lettre", in the example.
The conjugation of 'rendre' is 'il a rendu" as well as "elle a rendu", that is, the gender of the subject does not matter. In the example we are considering, since the auxiliary verb is 'avoir', the object "le livre" could matter, but only if it precedes the verb. So, even if we changed "le livre" to "la chemise", "rendu" would not change to "rendue"
First "livre" is masculine but in that case it does not matter as the direct object complement is after the verb "avoir" But when the direct object complement is before the verb "avoir" you have to match the gender and plural: for example " la fille l'a rendu " (l' being the book) and " la fille les a rendus " (les being the books) and also " la fille l'a rendue " (l' being the jacket which is "la veste") and " la fille les a rendues" (les being the jackets which are "les vestes". I hope it helps.
I think your answer was marked as incorrect because you left out the word "back".
"rendre" is not the same as "donner" here; it expresses the idea of giving back, returning something, rather than just giving.
Restituer à quelqu'un ce qui lui appartient ou ce qui lui revient de droit
Renvoyer, rapporter à quelqu'un ce qu'on a reçu de lui et qu'on ne veut ou ne peut garder