Why "i have tried to sleep" is wrong?
That is also correct if you have a context. By itself it probably means "eu tentei dormir".
Even if it's accepted in some context (with help from some "continous" complements), it's better to keep it simple:
As Paulenrique said:
I have tried to sleep = eu tentei dormir
I have been trying to sleep = eu tenho tentado dormir
Wouldn't 'Eu tentei dormir' be more like 'I tried to sleep'?
That too. But you can trust it to be "I have tried to sleep" as well.