The Verb has to follow the adverbs or the Interrogatives.

I often listen to the music. - Gyakran hallgatok zenét.

I never go to the beach. - Sosem megyek a partra.

Where are you? - Hol vagy?

What is your name? - Hogy hívnak?

It is not wrong. "It is rivers that I hear often".

Usually I hear lakes, but often I hear rivers. - Gyakran folyókat hallok.

It is a little bit weird with "often", it would work better with "sometimes".

Or:

Often I hear rivers, other times I hear seas. - Gyakran folyókat hallok, máskor tengereket.

Anyway, "gyakran" refers to what comes after it.