Translation:I forgot to return the book to the library.
If the sentence were
Then it's wrong because we can only use こと in this form.
I think now because the nominalized phrase is followed by を and the verb forget, の should be used.
Check out this article. It is quite comprehensive:
Hmm, I'll check it out! My impression thus far from various things I've read is that の imparts a sense that the situation in the subclause is taking place at the same moment as the outer clause (which explains why it's always used with sensory verbs like 見る and 聞く) , while こと is used to refer to situations more abstractly, and so for instance always gets used with verbs involving communication or internal thoughts.
In this case however , perhaps の is appropriate even though we're referring to a situation that didn't happen because it was supposed to happen at the same time we forgot about it. I'm not sure.
Actually I am not sure neither. The problem is there are not so many native comments around.
In the article I linked, there is an example which the author thinks both こと／の can be used.
I don't know if there is any difference in nature about to know something and to forget something. I don't know if it would make a difference if it is positive and をis used instead of は. If you find something else, please kindly bring it up here.
I managed to dig up one of the things I'd originally read about this https://japanese.stackexchange.com/questions/1395/what-is-the-difference-between-the-nominalizers-%E3%81%93%E3%81%A8-and-%E3%81%AE
I asked some Japanese speakers on IRC, and the general consensus was that both の and こと are fine here (though こと sounds slightly more polite), and they agreed with my explanation that の is okay because the thing we forgot was supposed to have happened at the time we forgot it.
In this case "no" turns the verb "kaesu" into a noun. Then you operate on this noun with a verb. You can also translate it as "I forgot returning books to the library".
In short, when you need to express an action upon action2 you take action2 as a plain form and add "no". Then you add other particles depending on which are required.
While the general meaning is the same, your sentence would sound differently. The original sentence does not say whose books they were (even if we are sure they could only belong to the library); yours does not indicate the recipient of the books. Maybe (in your sentence) the speaker forgot to return the library´s books to a friend of theirs who had borrowed them from the library originally.