It is rude when you use it without a noun. For example: "Mój stary" (literally it is my old, but it means my father, not my old). It's rude. But there's no problem with "Mój stary ojciec" (my old father).
According to me, it can sometimes sound rude. While referring to an old lady, you shouldn't say "stara pani", but "starsza pani" or "pani w podeszłym wieku".
Stary could be used as 'budy, pal, bro, friend' So: Cześć stary, jak leci? means Hi bro, what's up? Talking the same way to somebody 40+ years old could get you in real trouble.