Actually - That's true for the grammatical ending -ого which appears in Genetive and Accusative Singular of masculine and neuter forms. But много is a counter word which does not change in any case, it will always stay много and therefore -ого is no gramatical ending but simply a part of the word. However, one should not mix up the word with "много" with "многие", a word with a similar meaning "many", which works like an adjective, so changes in different cases. But this is a different word which only exist in plural forms.
What if one said Ya chasto otdyxayu ( I rest often). There I can also add Ya redko otdyxayu (etc.) These being adverbs, they should work, right?