"Which books are they interested in?"
Translation:O které knihy se zajímají?
I'm also just learning but this is how it seems to work. For this sentence you have two possibilities, either "They are interested in the books" or "The books interest them". For the first one you have "They" as subjects so you have to use "se zajímají o", to say that they themselves are the ones interested in something: "O knihy se zajímají". For the second option "The books" are the subjects, since they are doing the action of being interesting, so you only use "zajímají". And who are they being interesting to? Them, so "je". "Knihy je zajímají".
just to note that google translate gives 'o které knihy mají zájem' = 'which books are they interested in' and 'o kterou knihu se zajímají' = 'which book are they interested in'. Is it random or is there any grammatical/usage explanation for the different forms depending on whether book is singular or plural in this case?
As others have said, Google cannot be relied upon, especially not for anything that's the least bit complicated. But, no, the constructions that Google presented have nothing to do with the characteristics of the object.
There are just several ways in Czech to convey the idea of being interested in someone or something, just like there are in English. We can say, for example, "I am interested in those books" OR "Those books interest me" OR "I have an interest in those books" in English -- they all mean essentially the same thing, ultimately, though that message is conveyed through different grammatical approaches. Czech has several similar options... and I won’t attempt to address them here! :-)