In "Rōmae studētis", the word Rōma (Rome) is in the locative case, which means that something is being done in or at a place or location. For "You study Rome", you would need the accusative case, since Rōma is the direct object (the thing being studied): "Rōmam studētis".
Also keep in mind that "studētis" is used when you're talking to more than one person. If you were only talking to one person, you'd say "Rōmam studēs."
Romam studetis is wrong.
It would be "Romae": Romae studetis (dative)
Romae is both the locative, and the dative (needed with the verb "studere")
So, this one is tricky, and ambiguous, as the translation could be "You study in Rome" or "You study Rome" (as the topic of your study).
Even if the more obvious is "You study in Rome".
I reported "You study Rome" because it's not accepted yet, and it should be.
Locative, because it's the name of a city/town.
Here is a plain-English overview of what the cases are and how they work:
Latin cases, in English
Adjectives must agree in gender, number, and case with the nouns they modify, but they have their own declensions. Sometimes you get lucky and the adjective just happens to follow the same declension as the noun, but that is not a guarantee.
Yes, "Romae" is both the dative and the locative form of "Roma". Apparently the course contributors didn't think of that, because the accepted answers are all "... in Rome". You can always flag "You study Rome" after it marks you wrong and select the report option "My answer should be accepted."