It's important to know that timore is "fear" in the sense that "I fear we won't get there in time. Emotional fear is paura. The opposite of il timore is la speranza (hope) but the opposite of la paura is il coraggio (courage).
WordReference has a nice explanation.
In the light of that, I think the best English translation is probably "I just have one worry."
Using both words, one and single, while common usage in American English, is actually not proper grammar. Rather, it is redundant. I understand it would be considered emphasis, but it is still redundant. "I have a fear." simple statement. "I have one fear." emphasized without redundancy. Sorry. Yes, retired, but after teaching over 40 years it just doesn't go away.