Both 试一下 and 试试 means have a try, but 试试一下 seems weird. 试一试 is also fine.

Yes, repetition of a verb does mean "do something a little bit", but then you can't add "一下", which would be redundant.

Either 你想试试吗 or 你想试一下吗. You can't use both.

试试一下 is not usually said in Chinese, so combining repetition and 一下 does not happen (as this sounds redundant and unnatural).

I have a similar question. My answer - 你想试试吗 - came up incorrect. Don't know why.