Costar is a verb and verbs get conjugated. They don't change for gender. The subject here is "la mantequilla", a singular 3rd person, so the conjugational form is cuesta.
Saying that cuánto refers to cuesta is a bit misleading. Cuánto is a pronoun here, being the direct object of costar ("It costs this much money"), and Spanish pronouns generally take on the masculine form.
In "¿Cuánta mantequilla tienes?" the question word cuánta is functioning as an adjective, describing the amount of butter. Since the associated noun, mantequilla, is singular-feminine, the adjective will also be in its singular-feminine form.
In "¿Cuánto cuesta la mantequilla?" the question word cuánto is not associated with a noun anymore - it's a pronoun here. We're not asking about "how much butter", but just "how much".
This is not how you express the price of something in English. And it's farther away from the original Spanish sentence, where cuesta is indeed a verb form.
You could say "What is the price of the butter?" if you want to use a noun. In that case, the closest Spanish translation would be "¿Cuál es el precio de la mantequilla?"
Even though it's possible, it would be a bit pointless to ask about price as a property of butter in general.
The definite article can have two different functions. It can represent a specific item or a specific portion of something, just like the article "the" does in English. So "How much does the butter cost?" is still a valid translation.
The definite article can also be used for a generalisation. (I wouldn't say "butter in general", since that phrasing might be a bit misleading.) Generalisations are statements that you make about something, and you claim that that statements holds true for all instances of that thing. Like "Grass is green", which basically means "Whenever there is grass, it will be green."
Applied to the butter sentence, it will amount to "Whenever there is butter, how much will it cost?" I.e. you're asking about a single price for every portion of butter in the world. It's possible to ask this, but very unlikely to get a proper answer.