Because the way this sentence is constructed in Russian can be translated as: ' It is hard to believe to my words'. If you were to use nominative case, 'the words' would be the subject: 'my words believe hard'.
Would моим словам трудны верить be incorrect? "My words are hard to believe" implies that "hard" is modifying "my words", which is plural. But трудно also makes sense for the english equivalent "It is hard to believe my words".