Why isn't it "Je me perds touts les temps"?
If temps is plural why not tous les temps
I dont think that its actually plural - 'temps' looks like a plural noun but is actually a singular masculine noun.
Why is this perds and not perdre?
Because it is conjugated in the first person - not infinitive.
Would 'Je me perds tout le fois ' be correct?
No. There's "no tout le fois". Tout le temps is an idiomatic expression and is so fixed.