Vowel Sounds in Turkish and Vowel Harmony
There are basically 9 different vowel sounds and there are 8 letters that represent them in Turkish. It's a little bit hard to understand some of these sounds since they don't have any equivalent in some languages. I prepared an image that shows how to utter each sound by showing how the tongue and the mouth are shaped when we pronounce them.
As you see your tongue could be basically in two different shapes: forward and backward. That's actually why we categorize these vowels as front vowels and back vowels.
Beside your tongue, the shape of your mouth or your lips is also determinant when you utter a vowel sound. Your lips could be rounded or unrounded, and also closed or open.
Vowel harmony is based on the practice of keeping your tongue in the same position but changing the shape of your mouth. In other words when vowel harmony is involved the position of your tongue must always be the same but you have to reshape your lips/mouth according to some patterns for the next vowel. There are two different patterns in Vowel Harmony: 2-way and 4-way
2-way vowel harmony
What you do here is keeping your tongue in the same position while opening up and widening your mouth.
Some examples with -lAr (plural) ending
araba -> arabalar
gün -> günler
giysi -> giysiler
4-way vowel harmony
All you do is closing your lips while keeping your tongue in the same position
Some examples with -I (accusative) ending
adam -> adamı
göl -> gölü
güneş -> güneşi
herkes -> herkesi
Exceptions in Vowel Harmony
The vowel harmony itself doesn't have any exception but some words are pronounced in an exceptional way which leads the vowel harmony to work in an unusual way as well.
For example the word 'saat' is one of these exceptionally pronounced words and it works in the opposite way under vowel harmony. So the dative case of the word saat would be saate instead of saata. Let me explain why...
Some consonants such as d, t and l are pronounced using the tongue so since your tongue could be in one of the two shapes (forward and backward) these consonants have slightly different sounds for each of these shapes. It is usually determined by the previous vowel, that is to say if the previous vowel is a front vowel such e you pronounce the consonant keeping your tongue in a forward shape. So this is why the t sounds in the end of the words 'et' and 'at' are actually slightly different.
However some loanwords in Turkish have a fact called palatalization which overrides this usual way and requires you to shape your tongue in a forward way when uttering these tongue-related consonants (d, t and l) no matter which vowel comes before them. For example the word 'saat' is one of them and although the vowel a is a back-vowel you pronounce the t consonant at the end using your tongue in a forward shape as if there was a front vowel before it. Therefore the t sounds in the end of the words 'et' and 'saat' are actually the same.
So how does this affect the vowel harmony? Well you must not forget that the vowel harmony requires you to keep your tongue in the same shape. So when you utter the word 'saat' the final shape of your tongue would be forward because of the palatalization in the t sound at the end so this is why you should keep your tongue in the forward shape and use a front vowel for the next sound to stay in harmony. So the accusative case of the word saat would be saati instead of saatı.
I can say that these kinds of words are not quite many, at least they are few enough to memorize.
Sources of the images
Different tongue positions for vowels: http://thevoicenotes.com/choral-distortions-in-pursuit-of-blend-part-1/
Lip sync: https://theartofstopmotion.wordpress.com/2013/02/18/readings-on-lip-syncing/
It really depends on where we borrowed that word from. Rol comes from French and it's original pronunciation has palatalization so we secure that feature. But futbol comes from English and the l at the end of it is not palatalized in English either. If we had borrowed the word rol from English (role) it would probably have not had palatalization (you would have said rolu, rola).
Just to add: sometimes we borrow words from English and treat them as if they were French. For example "full" → "fullemek". (and not fullamak). Or for example in Ankara, there is a shopping mall with the name: AnkaMall. Everyone says: "AnkaMall'de" ; "AnkaMall'e gidelim" etc.
That's a very good point! But I would slightly disagree with the second example. Many people say it like that but there are also many people say it in "the correct way" as AnkaMall'a gidelim. When I check the official website of AnkaMall I see this sentence: Hayatınızı zenginleştirecek her şey ANKAmall'da. I think it is something that people gradually get more conscious of it.
Yet I agree that there is certainly a tendency to think like all these words are pronounced the way as if they came from French mainly because we have lots of French words like this such as festival, metal, rol, hol, karambol and many more that I can't count.