Bøker is the indefinite plural form of the noun, so you'd have to put the possessive before it:
mannens bøker ("the man's books").
When you use the definite plural form, you put the possessive after it:
bøkene til mannen ("the books belonging to the man").
Edit: Replaced "o" with "ø" to reflect
bok's vowel change.