I understand you wouldn't say very gladly in English, but sehr means very and gern means gladly, at best. To emphasize gladly, we'd say most gladly maybe?
Can "Sehr gern!" be used in other ways, e.g. Schwimmst du gern? – Ja, sehr gern! (Do you like to swim? – Yes, very much!)? "Very gladly!" seems to me like a pretty limited English translation for the sentence.