Why is it that sometimes "dreji" can mean "actually", but sometimes it doesn't?
Because actually has two different meanings in English. When you're correcting someone, you should use drīvose. When you're talking about something that really happened, use drējī instead.
actually adv. 1. in the act or in fact --> drējī 2. in point of fact - used to suggest something unexpected --> drīvose