Translation:The button is big, but not round.
Do you mean this sort of thing?
"You haven't given me the money yet." "Doch, I gave it to you yesterday."
This kind of "doch" isn't really identical with the one that means "but" (historically, it likely is; but in actual usage, it feels like a word of its own to me). It's a "yes" that puts a negation right:
"You didn't do what I said." "Yes, I did." "Didn't you go there yesterday already?" "Yes, but I liked it so much I went again." "I wasn't there." "Yes, you were."
"Do you think they have a room for us? The hotel must be fully booked this time of year." "Yes, I'm sure they'll have a room left." (Here, it's not a straight negation the "doch" puts right, but it's more like making a point against a doubt.)
(Bonus note: for "Yes [, I did / it was, etc], but [still]..." you can also use "Schon, aber..." - or even both: "Wasn't this nice?" "Doch, schon. But I still say we shouldn't have spent all that money.")
I understood it in this way: if something is going not naturally, if it surprise you, then you use "doch". For example: "Er hat Hunger, doch er isst nicht". If it doesn't surprise you, if all is going naturally, you use "aber"". . Another way: if you can say "but it's yet still...", you use "doch".
The "drop down" suggestions don't refer to the specific sentence, they're more like a dictionary entry with several possible meanings that don't fit every context.
In this sentence, you can just go word by word: if the German sentence says "aber nicht rund", just stick with "but not round".
I can't think of an example where "nicht" would be translated as "is not", but let me explain it with "do not" (the third drop down option I see): "Nicht rauchen!" = "Do not smoke!" But while the "nicht" does correspond with the "do not", "do not" isn't an actual translation of "nicht". "Nicht" is just "not". It just happens to be idiomatic for German language to say, literally, "Not to smoke!" instead of "Do not smoke!".
You’d use sondern when you’re saying that the subject is one thing -instead- of something else. Das Wetter heute ist nicht warm, sondern kalt, for instance. It can’t be both warm and cold; it has to be one or the other. Or Er ist kein Arzt, sondern Mechaniker.
Whereas with the button in the sentence here, it can be large and round, or large and not round, or not large and not round. Its size and its shape are not mutually exclusive. I might say that if I needed to find a button for my coat that is large and round. My friend looks in the button jar and says, “Here’s a button that might work. It’s large, but not round.” Der Knopf ist groß, aber nicht rund.