In Australian, there is quite a nuance of difference. between to and with , when used in context such as this.
Perhaps it is because of historical past, I muse.
When you talk to someone, it is that you giving them an instruction. You are person who is in authority, and you are giving them information or a directive. It is not a two way conversation. At least in most situations. While you would be understood, if you used the term to, it can inappropriately escalate a situation, or show to the person you are speaking, that you consider yourself of a different status, a status above the other person. The term to may also be used when it is that someone is accusing yourself of something, such as in a witness/legal court. And you are asked if you had a conversation. You might be likely to respond "I was talking to a man", or "I was talking to that man." Though you may equally as likely use the word "with" in this situation. Maybe part of that is that you are feeling like a fish out of water, and may be trying to find some skerrick of authority in an unsavory situation.
In short, it is better to avoid saying the sentence "I am talking to a man." Unless it is that you are wishing to express that you are a person of authority in a situation. If you are not a person of authority, you could find yourself in an unpleasant situation. And if you are a person of authority, you can still be more respectful and say "I am talking with a man", which if you show respect to others, at least in Australia, you will find that others will more likely be respectful also to you. And in my travels in other English speaking countries, I have also found this to be so. (ps I have also traveled many times to England, and also have English friends who live now in Australia, or in France. Which is yet another story ;P )
To have a talk with another person, is an indication that it is a conversation. Being a two way conversation. A conversation that you see yourself to the other person as an equal. That it is a two way exchange.