As in "go for a walk"?
It includes "going for a walk" but usually, it's more like going around, purposelessly without a destination.
"With whom do you wander?" is grammatical and correct and should be accepted. (I didn't try it.)
"WHOM" is correct, like it or not.
With whom are you wandering?