You only use "of" after "opposite" if you are contrasting the nature of 2 things. E.g. "Light is the opposite of darkness".
If you are talking about comparing positions, as in this sentence you don't use "of" after "opposite".
When speaking about direction of travel, you might use the word "to" after, for example: "The cars are going in the opposite direction to each other"