Does this sentene also mean "we belong to these books" ?
No, it doesn't mean that.
Kami milik buku-buku ini, would be that way of saying it.
Your sentence has different meaning. "We have these books" loosely mean "these books belong to us" (if you change the structure and verb), not otherwise.
I wrote "We have the books" and it marked wrong. Is it wrong to use "the" instead of "these"??