I've not encountered "vám" yet. What does it mean?
"to you" or just "you" as an object (in this sense https://en.wikipedia.org/wiki/Object_pronoun)
As VladaFu stated, it means "to you." This is because it's "vy" (you) in the Dative case, which you can think of as "indirect object" case.
Is there a rule when thanks is Dekuji vam and when diky?
Děkuji is more formal, díky is less formal. Děkuju is in between.
uh. wasn't "díky" thanks?
i belive "děkuji vám" is more like "thank you very much"?
Yes, "Thank you", the "very much" is probably extra and we have other equivalents in Czech "Děkuju pěkně, Děkuji mockrát, Díky moc,...). We chose not to distinguish between the level of formality and we accept all other equivalents everywhere.
So are there any differences between the two thanks?
Read the reply to AnnaPeters and the rest of the discussion.