That is also correct if you have a context. By itself it probably means "eu tentei dormir".
Even if it's accepted in some context (with help from some "continous" complements), it's better to keep it simple:
As Paulenrique said:
I have tried to sleep = eu tentei dormir
I have been trying to sleep = eu tenho tentado dormir