You can be in the middle of the city center, or at the edges of the city center. You can be well in the middle of a city but not in its center (often happens with coastal cities - the physical middle of a city due to growth shifts away from the coast, yet the city center - the focal point of activity and heritage - remains close to the coast).
Is this just for populated areas, or anything with a center, like center court/field, basketball center position, center of attention, storm center, shopping center, self-centered, political center, job center...?
And the related word, central which is used for many main train stations, especially in Europe, or as in central focus, central character, central office...
The town center (where the most activity is) in Portugal is often called Baixa and downtown seems to really only apply in the Americas (I would be hard-pressed to figure out what the "downtown" is in Istanbul, Paris, and London for instance).