This idiom occurs only a few times among the ancient authors, and as far as I can see always in the context of army leaders having formal parleys or negotiations over war and peace.
I would not advise using this to mean something like "have a chat", as it seems to be being used here. It is probably better to think of it as "to parley", "to enter into discussion", "to engage in negotiations".
"Colloquium habere" is found, as far as I can tell, only once among the ancient authors, but once is enough, I suppose.
Other ways to say "to converse": "colloqui", "colloquium/sermonem serere", "sermonem conferre" "sermocinari", "confabulari", "sermonem caedere".