I think this is just one of those ideas that's baffling to many British people. I like the challenge of investigating different concepts and I can remember enough Duoese to get past this one!
"I will let you go... " that sounds as though the other person has just said" look, I have to go now." More likely is "I'm going now... " - the French implies the speaker is the one going off and the English should do the same.
Nobody is leaving anyone else. They are on the telephone in different places. "I called, imposing upon you, and have taken up enough of your time, so I'll let you go for now."
This would translate better as "I'm off, but I'll call you back later"