Actually it's only mandatory when the direct object pronoun is either lo, li, la or le. With mi, ci, ti and vi you can leave it as -ato, -uto or -ito. I don't understand why it's mandatory with li and le. For lo and la, I understand it because those are the two pronouns that join onto the avere verb. In this case, to form l'avevo. We can't tell by looking at l'avevo whether that was 'lo avevo' or 'la avevo' so changing the ending of the verb makes sense, so we regain the information we lost when the avere word 'absorbed' the pronoun. But we never join 'li' or 'le' onto the avere word so there's no information being lost, and yet you still have to change the verb to agree with them. Always thought that was strange. In any case, those are the only 4 direct object pronouns where the verb has to agree