Translation:There are more houses in this town than there are people.
«Бо́льше» is a comparative degree of «мно́го» and requires the same grammar.
If that helps, you could imagine «бо́льше» to mean 'a larger number'. Of course, Duolingo won't accept 'a larger number' (that would correspond to «бо́льшее число́»), but it helps to understand why Genitive is used: Genitive is often translated with 'of', so «бо́льше домо́в» ≈ 'a larger number of houses'.
N. B. «Бо́льше» has several meanings, and 'more (~a larger number of)' (comparative degree of мно́го) is only one of them. Notably, it can also me a comparative degree of the adjective 'большо́й', and then it's translated 'bigger, larger'.
It can be genitive singular if the noun is uncountable:
- бо́льше воды 'more waters',
- нет воды́ 'there is no water',
- отсу́тствие воды 'absence of water'.
«Не домов» wouldn't normally mean there's a zero amount of houses, it usually means 'not houses', 'something else than houses' in genitive:
- Э́то бы́ли фотогра́фии не домо́в, а люде́й. 'Those were photos not of phouse, but of people'.