In the notes it says this the question form, so shouldn't the translation be "were you"?
This is a zombie - it was deleted sometime ago during beta testing, but due to problems with Duolingo systems it has not yet vanished.
It is a sentence fragment that does not work on its own. Until it gets deleted just give whatevrr answer it seems to need, either 'you were' or 'were you'.
Yes, you're right. This is a weird phrase Duolingo is asking you translate. I suspect the problem is that they're asking you to translate a fragment of a larger sentence. (You'd never see o't ti'n on its own with the 'n.)
So it could be from, say Beth o't ti'n wneud? (What were you doing?) or maybe Do'n i ddim yn siŵr beth o't ti'n wneud (I wasn't sure what you were doing). Your guess is as good as mine.