Wouldn't "not handsome" be more accurately described as "ne bela", since "ne" = "no/not", while "-mal" = "opposite"?
(Personally, I would say "ugly" is closer to being a complete opposite of "handsome", than "not handsome" is? Especially considering that we have a different word for "not" already?)
You are so right! So we are saying "The man is the opposite of handsome." It still seems slightly less harsh than "The man is ugly." One has to think, now what is the opposite of handsome, before one gets there? I think by that time the speaker might sneak away. It really isn't any better though. I would rather say "She is the opposite of tall."