Well, this is a "Not A but B" type of sentence. And remember, what is after "Nem" is the thing that is being negated. If you take "mi" out of that position, it is not negated anymore. You are changing the whole sentence.

The sentence above says "Not us but them".
'A' is "us"
'B' is "them".

Your version would be "(We are) not talking on the phone but ..."
But what? "Not talking but them"? 'A' being "talking on the phone"
'B' needs an equal alternative.
So, 'B' needs to be changed.
And the structure of the sentence is also better if slightly changed:
"We are not talking on the phone but they, on the other hand, are (talking)."
"Mi nem telefonálunk, ők viszont telefonálnak."
The second verb, as it is a repeating one, can be replaced with the word "igen" (yes). That is how Hungarian does it:
"Mi nem telefonálunk, ők viszont igen."

Of course, this is a totally different sentence than the one above.