Well, yes and yes. Fuori can translate to off in other scenarios without tema but they too are rather idiomatic: to go off road / andare fuori strada, to 'off' someone / fare fuori qualcuno. Remember that prepositions don't always translate the way you'd expect them to. Learning to use them requires practice and memorization.
Oh, are you not a native English speaker? off topic means you are no longer talking about the main, or original, topic in a conversation .. your conversation has drifted to talking about something else. Now I finally understand the Italian sentence ... 'sono fuori tema' literally is 'I am outside the theme' which is essentially what off topic means.