I see a difference there:
(I'm) sorry (that) I didn't listen to you. (I haven't acted in accordance with your advice) - Wybacz, że cię nie (u)słuchałem.
Sorry, I wasn't listening to you. (I haven't been paying attention while you were talking) - Przepraszam, nie słuchałem cię.