Why नीलें और पीलें किताबें is wrong? Several blue and several yellow books are may be meant by "Blue and yellow books".
किताब is a feminine noun. So, you must always use feminine forms of the adjectives.
It's नीली and पीली for feminine plural as well as singular nouns. नीले और पीले are only for masculine (or mixed gender) plurals.