In the [noun noun] phrase "tomato sauce", the first noun, "tomato", is descriptive, telling us what kind of sauce it is, not how many tomatoes are in it; that's how it's said in English. The same is true for tomato juice, tomato pie, tomato salad, egg salad, strawberry pie, grape juice, apple tree, bus stop, train station, et cetera.
The phrase "tomatoes sauce" is wrong because it is not idiomatic English.
The phrase "tomatoes' sauce" is wrong here because the apostrophe makes the first noun possessive, telling us that the sauce belongs to, or is owned by, some tomatoes.
Also, you should not assume that Hebrew always uses the plural of the descriptive ingredient. Although it does so here, and in other tomato products such as tomato juice, "מיץ עגבניות", and in some other juices such as orange juice, "מיץ תפוזים", there's also carrot juice, which is "מיץ גזר".
jg9000 already answered your first question.
Walrosse's spelling of קטשופ is without dagesh for the same reason that it's without vowels: that's typical for Hebrew. But you can always include any of those marks when you feel that it will help you or the person with whom you're communicating: here, have some קֶטְשׁוֹפּ. (However we are not supposed to use those marks in answering Duolingo's lessons.)
Wanda, the phrase "רוטב עגבניות" is a compact way of saying "רוטב של עגבניות", literally a sauce (singular) of tomatoes (plural). So the overall phrase is grammatically singular: it's just one sauce.
It's a Hebrew noun-noun "construct phrase" called a "smichut" ("סמיכות"); you can read about that in the course notes at https://duome.eu/tips/en/he
and you can find other sources online and in textbooks.
Note that the first noun in a smichut determines the gender and number of the smichut, regardless of the gender and number of the second noun. The smichut "רוטב עגבניות" is grammatically singular and masculine because רוטב is singular and masculine, even though עגבניות is plural and feminine.
Having said all that, the second noun is not plural in every such smichut. E.g., while orange juice is "מיץ תפוזים", carrot juice is "מיץ גזר".