their is их. It does not have any forms. Generally, 3rd person possessives are the same as the Genitive case of the corresponding pronoun (его, её, их).
There is no rule that masculine words should not end in Ь. It is still a consonant-ending masculine noun.
Statistically speaking, feminine nouns are the majority amongst nouns ending in a ь in Nominative, comprising almost 70% of them.