Is this about a book that is historical (say, from 1700), a book that was a great success (a historical one), or a book that deals with history?
Any of the above. The Irish version is just as ambiguous as the English one!
I would have thought 'leabhar stairiúl' referred more to a book that is historical and 'leabhar staire' (using the genitive) would be a book about history.
If a book about history is one correct interpretation then why does it mark 'a history book' wrong?
Because "a history book" is "leabhair staire".
In English, the term "a historical novel", usually refers to a novel set in a particular period of history, whereas "a history book" deals with the subject of history - it may cover a particular incident or period of time, or give a general overview of a long period of history. While a "leabhair staire" can cover both of those terms because of the way the genitive is interpreted (a book of history), "leabhar stairiúil" uses an adjective, and really only means "historical".
The first thing I thought when I read the new word:
Gáirim go stairiúil!
Does this happen in Irish, too? xD