"Mom does not have a brother."
Translation:У мамы нет брата.
These are forms of the same word. «Мамы» is the Genitive one.
У is the prepositions that always wants the Genitive (regardless of the exact meaning), so you use it here. Using the у+Gen (+ «есть») phrase is the most common and ubiquitous structure to express possession in Russian, so we have it early on.
«Не» generally corresponds to "not" and we use it to negate a member of the sentence. «Нет» usually means "no" and we use it to start a negative answer or to express a lack of something. Нет can work as a short negative answer (Ты спишь? — Нет.). Alternatively, if you are answering a question that has a verb, you can use не + the form of this verb that you would use in a dull fledged sentence (Ты спишь? — Не сплю). Of course, we do not have dialogues in our course.
We also use нет when omitting the part of the sentence that gets negated (but would it be repeated if it were left). For example, "Я переводчик, а моя жена нет" (I am a translator and my wife isn't) or "Они спят, а мы — нет" (They are sleeping and we aren't).