From what I've gathered just from doing this course (and I could be wrong), "olika" seems to be used when talking about things in an "unalike" qualifying relationship to one another, as in "Han och hans bror är olika"; whereas "annan" is used when discussing things that are in opposition to one another, as in the case above where he buys a completely separate suit from the first choice that the sentence implies to exist ("Han köper en annan kostym, inte den första").
A much simpler way to think about it in my opinion is that if you can't replace the word "different" with "unalike" when speaking the same sentence in English then "olika" would apply, not "annan"
I'm a native English speaker..... "another" does not mean a different one........ that would be "an other". "Another" means "additional", not "different", although many English speakers would probably argue this, but it's a common spelling mistake.
-I wanted 2 suits, so I bought another one.
-I didn't like the first suit, so I looked at an other one.
Pronounced the same, but the spelling is not.