"I want to go to my garden!"
Translation:Je veux aller dans mon jardin !
Usually, you hear the conjugated verb first, then the infinitive one (if any)
I do believe allez and aller sound the same, I am not a native though, so I'm not sure about it.
Just need to train your ears and know the basic rules of sentence constructing, then you should be fine :)