See my explanation here.
The pronoun has a 'front part' and an 'ending'. The 'front part' is chosen based on who the thing belongs to: sein- means the thing belongs to a male/neuter person/thing. The 'ending' is chosen based on the gender and case of the thing itself: -en is required for masculine nouns (such as Hund) in accusative case.
Other possible sentences to illustrate this:
Sie mag seinen Hund = "She likes his dog"
Sie mag seine Katze = "She likes his cat"
Sie mag ihren Hund = "She likes her dog"
Sie mag ihre Katze = "She likes her cat"
Er mag seinen Hund = "He likes his dog"
Er mag seine Katze = "He likes his cat"
Er mag ihren Hund = "He likes her dog"
Er mag ihre Katze = "He likes her cat"
Ihr mögt unsere Kuchen = "You like our cakes"
Don't be confused by the fact that the pronoun doesn't match the subject - in German as in English that can be ambiguous if they match, but they obviously don't have to:
"He likes her cat" = He likes the cat belonging to some other woman.
"He likes his dog" = He likes his own dog, or he likes the dog belonging to some other man.
Why seinen and not sein Hund?
Becase Hund is masculine and is the direct object of the verb mögen here, so it's in the accusative case.
Thus you need masculine accusative seinen here.
sein would be masculine nominative (wrong case) or neuter nominative/accusative (wrong gender).