I don't know... maybe! A lot of these idiomatic expressions don't have an obvious literal explanation.
I actually can't explain it very well at all. There are situations in which "by x" WOULD seem more natural to me, but I can't figure out what determines it!
It may even just be a regional variation thing. It's such a minor shade of meaning.
In Ukrainian one can shuffle the word positions almost freely to achieve the desired emphasis on the words you like.
The variants of this same sentence would include:
"Куди ти їздив на машині вчора?", "Куди ти їздив вчора на машині?", "Куди ти вчора їздив на машині?"
"Куди ти вчора на машині їздив?", "Куди ти на машині вчора їздив?", ...
"Ти куди їздив на машині вчора ?", + variations
Just SO MANY. This freedom of shuffling 1) assists what we have in every language with intonations: "Where did you drive yesterday?", "Where did you drive yesterday?" 2) gives one time to think of what to say next while already saying what you know first 3) leaving the most exciting part to the end ("I like hamburgers... not!") 4) urgently saying the most important part first and then adding details 5) ...