Are there many irregular formation of plural nouns? I thought that I simply add at the end "-и" and it is a plural noun. But plural of "кіт" is not "кіти", but "коти".
It is the regular formation.
о/е often replace і when a closed syllable turns into an open one, so it is actually a rule typical of many words rather than an exception.
One should bear in mind that it happens often (as you said), but not always: ліс - ліси
Мій кіт + моя кішка = Мої коти? Or only мої коти = мій кіт + мій кіт?
both options are correct