Is it possible to know whether it is o, a, or á before the t after the a or do you just have to remember?
If the noun ends in -a, it's lengthened before the -t: vacsora -> vacsorát; kutya -> kutyát (it works similarly when the noun ends in -e: kecske -> kecsét; if the noun ends in another vowel, there's nothing to add or change: kapu -> kaput)
And as for "o" and "a" there's no rule I'm afraid... Both are link vowels used for back-vowel words. The basic one is "o" (kalap -> kalapot; robot -> robotot), but there are a few exceptions that take "a" instead (had -> hadat; madár -> madarat; hó -> havat, etc.). As you can see, in the case of the last two examples even the stem changes.