Natural English would be either "We took each other's books outside" or "We removed each other's books and took them outside." The verb "remove" doesn't appear to be able to be used with the destination as well, so that's why this sounds really odd in English.
-toleana hadithi, 'tell each other a story' is a definition from Kiswahili: Msingi wa Kusema, Kusoma, na Kuandika by Thomas J. Hinnebusch and Sarah M. Mirza. So could this sentence possibly mean "We gave each other (exchanged) books outside?" Also, I can't recall for certain, but it seems to me I might have heard or seen the phrase "-toleana zawadi" (give each other presents) somewhere.