Without any context I'd say κούπα is a mug-cup, but φλιτζάνι is smaller, like a tea cup. However, a lot of the time you hear "κούπα του καφέ" or "φλιτζάνι του τσαγιού" when talking about quantities, and it seems to me that "κούπα του καφέ" is a bit ambiguous as it could be the size of a tea cup too, for cappuccinos etc. :) But φλιτζάνι is definitely smaller in any case and possibly finer or more elaborately decorated than a κούπα.
What is the defining criterion of a κούπα as opposed to a ποτήρι ? Does a κούπα have to have a handle?
Yes ;) Colloquially, "κούπα" is also used to refer to trophies in sports.