Uz chapu, co se prihodilo. Doporucuju si nastudovat anglicky slovosled pro prislovce, ceska jazykova intuice nas casto vede spatne. Treba https://dictionary.cambridge.org/grammar/british-grammar/adverbs-of-time-and-frequency/hardly-ever-rarely-scarcely-seldom ale i mnoho dalsich clanku o "adverbs, position, sentence, word order, ...".
Cituji: In more formal styles, we put them in front position and invert the subject and verb. If there is no auxiliary or modal verb, we use do/does/did.
Not sure what this is worth but a native english speaker would say 'I seldom understand you' or 'seldom do I understand you'.
Your sentence, 'seldom I understand you' is completely understable and I cannot tell you what it is that's wrong about it but it's at least unnatural sounding and I suspect there's something incorrect about it.
"rarely" is already accepted.
"seldom" is correct, used more often than and preferable to "seldomly". It's often considered bad style to use "seldomly", since the "-ly" ending is redundant.