Yes, if it was something singular (like a shirt) then "sort" would be used in this sentence (Min skjorte er sort).
Also, the "e-form" of a plural is also used when an adjective comes between the definite article (English "the") and the noun, or a possessive pronoun and a noun, even when the noun is singular. For example:
"Skjorten er sort" = The shirt is black
"Den/Min/Martins sorte skjorte" = The/My/Martin's black shirt
"En sort skjorte" = A black shirt