It is grammatical English, it's just that nobody says it precisely like that because it sounds too stilted nowadays; it evokes Shakespearean times. Imagine yourself at a conservative book club. Somebody suggests "How about '50 Shades of Grey'?", and half the room replies "Let's not and say we did".
"Let's <do something>" is fairly common usage, as is "Let's not <do something>". Let's not go out tonight. Let's read 50 books this year!