When to use جارِك with line under 'Ra' or Hamzah and when to use it with line above 'Ra' or Fathah?
The line under the "Ra" is called Kasra and is used when you're talking to a female. When you're talking to a male, you use the Fathah.
Is "ya" necessary to form the vocative, or can it be left out? Will Rania know I'm addressing her if I just say "Rania" and not "ya Rania"?
in most cases it is necessary in standard Arabic.
In this case if it is left out it would read:
جارك عمر دكتور رانيا
An Arab would hear that as: Your neighbor Omar is Rania's doctor.
you can say
رانيا، جارك عمر دكتور