How do I discern when I should translate an imperative more literally (i.e. "Go to work"), and when I should add "Let's"?
In imperative, you conjugate the verb according to who you're talking to--"tu", "vous", or "nous". This one is in the "nous" form, so it makes sense to translate it as "let's". An order to someone else would be "allez au travail" or "va au travail". Note that it's "va" without the S, because there are some funny rules for "tu" in imperative. This page explains more: http://french.about.com/od/grammar/a/imperative_2.htm
Edit: fixed 1st person mistake
Is it just me, or does the computer voice say this with an air of resignation and depression?