Кафе is a loanword, so it's pronounced «кафэ».
Э is only consistently used after vowels and in the beginning of words. After consonants, Э is often replaced by Е.
This happens to loanwords: интерне́т 'Internet' (pronounced интэрнэт), тест 'test' (pronounced тэст), стелла 'monument' (pronounced стэлла), ге́незис 'genesis, origin' (pronounced ге́нэзис, colloquially генэ́зис).
In newer loanwords, both Э and Е can be written: both фле́шка and флэ́шка are possible to write 'USB flash drive'. In older loanwords, only Е is considered correct (it's a mistake to write кафэ even though it's pronounced this way).
However, many loanwords are naturalised in Russian, and then a normal Е is pronounced: текст 'text', сема́нтика 'semantics'.
So, when seeing a Е after a consonant in a loanword, there's no real way of knowing its pronunciation. Luckily, Russian speakers don't know it, either. :D We have a lot of variation, so some people would pronounce квест 'quest (type of games)' as квэст, while others would say квест. So, if you pronounce those words as they are written, it would be perfectly understandable.