You're absolutely right: no native speaker of English would be confused by the meaning but we try only to accept grammatically correct answers because native English speakers are not the only people using this course, and adverbs of frequency are not usually placed at the beginning of sentences.
'Rarely' at Cambridge Dictionary - That is a British English grammar reference but if you look at the example sentences for 'rarely' on Merriam Webster's site, for a US example, you will see they follow the same rules, including the subject/verb inversion for formal effect.
Well... it kinda says it's on foot, but "chodzić/iść" (which suggest going on foot) frankly are also used if we consider it completely irrelevant whether we walked there or drove there :D So as usual, nothing is easy with Polish, right?
To make it 100% clear that you mean "on foot", you could add one of those three: "pieszo/na piechotę/piechotą".
Although a sentence "Rzadko chodzicie do parku, ale często tam jeździcie", so your sentence in brackets, also makes sense and the contrast gets rid of any ambiguity of "chodzicie".