"Ich treffe ihn fast nie" would never be "I almost never hit him" because "treffen" in the sense of striking implies hitting the mark. So if you shoot him but miss the mark you can say "Ich schieße die ganze Zeit auf ihn aber ich treffe ihn fast nie". But outside this specific scenario "Ich treffe ihn" means "I meet him".
As a native US English speaker, "to meet someone" does not imply a meeting but seeing in passing or be introducted (which would be very odd in this construction). Whereas "to meet with someone" does not imply to see in passing or to be introduced to but to have a meeting/conversation/etc.
Thus, to me they are very different meanings and are not at all interchangeable.
I would agree that "I almost never meet him" is less usual in American English than "I almost never meet with him," as we would say "to meet" as a one-time event, the first time you meet someone, and "to meet with" as a thing you can do over and over, but I would hesitate to say we'd never say that. Different people say things differently, especially in different parts of the world, and the context would make clear which meaning of "meet" was intended.