Uz chapu, co se prihodilo. Doporucuju si nastudovat anglicky slovosled pro prislovce, ceska jazykova intuice nas casto vede spatne. Treba https://dictionary.cambridge.org/grammar/british-grammar/adverbs-of-time-and-frequency/hardly-ever-rarely-scarcely-seldom ale i mnoho dalsich clanku o "adverbs, position, sentence, word order, ...".
Cituji: In more formal styles, we put them in front position and invert the subject and verb. If there is no auxiliary or modal verb, we use do/does/did.
Not sure what this is worth but a native english speaker would say 'I seldom understand you' or 'seldom do I understand you'.
Your sentence, 'seldom I understand you' is completely understable and I cannot tell you what it is that's wrong about it but it's at least unnatural sounding and I suspect there's something incorrect about it.