Quite correct, the English infinitive would be [to] come.
In Swedish, we have a verb for to usually do something: (att) bruka [some verb]. So the reason it isn't kommer is because brukar modifies the upcoming verb to an infinitive, but is itself in third person singular.
I'm not sure if there's a difference between Swedish and English here or not, maybe there is. I'm pretty sure it doesn't always imply that in English either though, think about how it could be used in a context of a story told in the past tense. And generally you probably wouldn't say Han brukade komma hem sent in Swedish either if he still brukar komma hem sent, then we'd say that instead. But maybe you're onto something.
Interesting, apparently Old English used to use "brucan" for this purpose as well, but when we adopted the word "use" from Latin via French, it also displaced "brucan" to give "use" a secondary meaning. That doesn't explain why there's this strange gap in the standard present tense, though... I wonder if it was some sort of parallelism with "use" that led "usually" to replace it in that tense.