This sentence could be either 'That is his book' or 'That is her book'. उसकी/उसका/उसके conjugates with the object, not the subject. The object is किताब here, which is feminine, so it is उसकी regardless of whether वह is him or her.
I think it's something you learn along with vocab. Native speakers don't seem to be able to give me a precise answer either. They just are what they are. If anyone has a better answer I'd be more than happy to be corrected?