"I do not sing songs."
は does not (necessarily) indicate the subject! It indicates the topic, which may also be the direct object. If the direct object is the topic, the は replaces the を.
Would probably be good to cover this more thoroughly somewhere in the lessons. I think it's by far one of the most confusing things in the language
That translates to "I don't sing." Depending on the context it could mean the same thing, but this sentence wants you to specify songs as the things you specifically don't sing.
"うたいません。" is the most natural answer. I suppose you could say "I don't sing songs" doesn't necessarily mean "I don't sing" since you could sing gibberish that you make up and claim it's not a song. It's implied that you don't sing songs, so you don't have to say "I don't sing songs." At any rate, if you wanted to translate into something that's not quite native level you'd say "歌をうたいません。" Unless their teaching point is that the verb, to sing, in Japanese is intransitive; that would make the sentence much more difficult to translate. "を" is the object marker and songs is the object of this sentence.
How can you tell when to use は, か, を, に, or で as object markers or particles? I just need a way to categorize them.
See here for most of them: https://www.freejapaneselessons.com/lesson04.cfm
は is a topic marker. It marks the theme/topic of the sentence.
が (not か) is a subject marker. It generally marks the noun that is performing the action/verb.
を is an object marker. It marks the direct object of the sentence - the noun being acted upon.
に has a bunch of different uses; it can be an indirect object marker (e.g. the person you're writing a letter to), a direction marker, or a time marker.
で also has several uses; it can be a location marker (i.e. where an action/event occurs), or it can describe a method/means (i.e. the means by which an action is done or takes place). There are a few other uses (e.g. marking the material something is made out of, or indicating a sum), but these two are probably the most important to know.
I really suggest you to find some lessons on the web. Duolingo is a great tool but you can't use only this to fully understand the japanese grammar.