"He would have answered if he was home."
Translation:Han ville ha svart om han var hjemme.
In this context there is no difference, but both words have other functions in which they're not interchangeable.
"Hvis" can be a relative pronoun and a question word.
"Om" can be used in expressions of time, in hypothetical comparisons, and to mean "(a)round" or "via".