Translation:We found him in the zoo, with the elephants.
That sounds more correct to me. "... with the elephants." sounds like he is in the enclosure with them.
"... by the elephants" sounds like he is near the elephants but not actually between them.
Although, if he was at a safari park rather than a zoo he might be "with the elephants".
Works totally fine as a sentence.
You rearrange the "narration" a bit, though. The original sentence sounds like "we found him at the zoo; to be exact, we found him with the elephants / at the elephant compound"; your sentence sounds like "we found him with the elephants that are at the zoo / at the elephant compound of the zoo". Doesn't make an actual difference, of course, just saying.