No, this is a common mistranslation by English speakers. You can't put the Korean for "no" and "thank you" together to make "no, thank you". Look on YouTube. There are lots of videos talking about these and give plenty of options.
They mean the same thing. The differences are more of a cultural/social thing... By using them in converstion and listening to them being used in conversation, you can pick up on the slight differences in connotation.
Another person explained this on earlier discusses but I will write it again, 감사합니다 comes from Chinese while 고맙습니다 is purely Korean. There isn't a big difference but I guess 고맙습니다 is a little more informal than 감사합니다 since it's like the longer version of 고마 워 which means 'Sorry' in informal language.
Romanizations are mostly confusing, it's originally romanized as gamsahabnida but it's pronounced as gamsahamnida. Try not to use romanizations as much as possible since it will make stuff more complicated and it won't help you to practice, really.
아니요 (and the shortened version 아뇨) is a more polite version of 아니. it's more polite than 아니 because it has the 요 at the end.