"The museum is on the right."
Translation:Το μουσείο είναι δεξιά.
Nope. If it was feminine, it would've been "στη δεξιά", no? ^.^
What I assume you are referring to is the adjective δεξιός, δεξιά, δεξί. Yes, δεξιά is feminine when it's used as an adjective (for exp. στη δεξιά πλευρά - on the right side) , but not in this case, because of that στα.
Στα δεξιά works just like the adverb δεξιά would, implying "προς τα δεξιά". ^.^