Because of cases.
ο καφές is the nominative case, used e.g. for the subject of a verb.
του καφέ is the genitive case, used e.g. in the meaning of "of the coffee" or "the coffee's".
τον καφέ is the accusative case, used e.g. after many prepositions or as the object of a verb.
The most reliable way is to look it up in a dictionary.
But you can usually guess based on the ending:
- Masculine: -ος, -ας, -ης, -ες, -ους
- Feminine: -η, -α, -ου
- Neuter: -ο, -ι, -μα
It’s not foolproof (the ending -ος in particular can be any gender) but that’s a good rule of thumb.