Isn't 걱정하지 for let's? Is it always needed to form the imperative? I'm very confused!
-지 말다 (+ appropriate verb endings) is the form you use for a negative imperative. You can think of it as a bit similar to English "refrain from..."
하자 is "let's"