No es expresión válida. If you want to use 'to' then it would be: 'You didn't allow me to walk'. The expression 'let me' is always followed directly by the verb. Hence, 'He didn't let me walk'.
'Let me' and 'allow me to' are (usually) interchangeable. 'Let me' implies seeking permission, while 'allow me to' is often used as a polite way to introduce something that you want to say or do. For example, 'Allow me to introduce myself'.