No, it would sound as if someone addressed to police. But actually it's an exclamation.
I was going to ask how many cases policja was the correct ending for actually, because there are a few that you could exclaim 'police' in; my first thought when I saw this was vocative, as by someone calling for any nearby police to come to them, which I think AtissEmiu may also have been thinking of, so thanks for confirming that that would instead be 'policjo'; so, I suppose this sentence is either in nominative, as in, "[Look, the] police [are here]!" (would that still be nominative in Polish?) or whatever case you would use if you were saying, "[I'm/We're (with/from the)] police!"/"[I am a / We are] police [officer(s)]!" - except now I think about it they would all be instrumental wouldn't they? Which would be declined differently... so "policja!" is for the second example exclamation I gave?
Vocative case, or Wołacz (one who calls, shouts) has a little different usage than it's name can suggest. It's used when someone is nearby and we are talking directly to him/her. When someone is far from us we use Nominative. Nowadays vocative is used more rarely than ever and it is replaced by nominative - this process is, from what I know, finished in Slovak, where only God himself retained different vocative. Besides computer programs cannot decline :(
So when you want to call the polis you shout: Policja.
And example of vocative:
Polska policjo, kiedy skończysz prześladować zwykłych ludzi zajmiesz się prawdziwymi przestępcami!
Polish police, when will you stop repressing ordinary people and start fighting real criminals!
Thank you, I never knew it could only be used in that way (I suppose I had kind of noticed it had started to stop being used that way as well).