The navy is the component of a military force on oceans and other bodies of water. To be "in the navy" means to be a member of that force, or less commonly can be a reference to the ways things are done in that force ("Why are the mechanics wearing white? Only in the navy!").
"In the navy" is a sentence fragment and not a full setence as the phrase lacks a verb. It's also a title of and a lyric from a famous song by The Village People.