Should this be understood physically (i.e. "pressing against") or metaphorically (i.e. "opposed to")? Or both?
Mostly used metaphorically.
Can be used in the physical sense of being opposite to me, but that's a bit archaic, one would usually used מול for that.
Why is "you against me" wrong? In the context of a challenge or a match, I mean...