"Había perdido sólo cuatro segundos."
Translation:I had only lost four seconds.
You are correct. There is also a ser + participle version of the passive voice in Spanish, but it's used relatively rarely. Reflexive passive is way more common. Se habla español aquí. Spanish is spoken here.
I put "I had missed just four seconds." Can anyone tell me if that's actually correct. I was thinking in context of perhaps getting distracted for a few seconds and missing something. That seems like it would be the equivalent of "I had only lost four seconds." which is the correct translation according to them.