Translation:What would you have answered if he asked for your password?
I'm still a little confused about using 'hvis' and 'om' for 'if'. Which one works better in which case?
"Hvis" is usually the best translation for "if".
"Om" is used to mean
"whether": "Kan du huske om jeg låste døren?" "Can you remember whether I locked the door?"
"if may": "Hva mener du med det, om jeg tør spørre?" "What do you mean by that, if I may ask?"
"as if" (with "som": "som om"): "Han løp som om det stod om livet." "He ran as if it was a matter of life and death."
and as a an admitting conjunction. "Han virket besluttsom, om noe alvorlig." "He seemed determined, if a bit grave."
Isn't this supposed to be "What would you have answered if he had asked you for your password?"? The english in this module seems broken to me
It's accepted as correct when you use your version. I think they're trying to stay as close to the Norwegian original as possible (even though it sometimes sounds awkward in English) but it helps you when you have to translate the same sentence from English back to Norwegian.