How would you say, my big hat?
mein große Hut ... oder...mein großer Hut
The latter, "mein großer Hut."
then what is the German for cap? or the cotton hat...?
Well you know, that really depends on what kind of cap or hat, there are Baseballcaps, Mützen, Hüte and so on. There are as many different words as in English.
How do you decide whether to use 'Mein' or 'Meine'? I googled and the results I found said that Meine was used for feminine nouns, but the answer for this one is 'Mein', which according to Google is for the masculine nominative case.
Exactly, you gave yourself the answer. 'Der Hut' is masculine.
But Hut is feminine !! so meine should correct isn't it?
Hut is masculine. The hint is wrong.
Ya, this needs to be fixed. It's confusing.