It's just the name of a case. It's often used to mean how something was done; in Russian, for example, the instrumental case on its own is used to mean "with/by means of": I ate with a spoon, I cut with a knife; it's the instrument by which an action was performed. This is not its only use, but I believe it's where the name comes from.
In several Slavic languages it is used with verbs of being or becoming. It seems like Polish uses it a lot. I am sure it will get explained in more detail later in the course. Don't worry about it!
The object is either in nominative orinstrumental case; the former is often used when introducing oneself, and implies the object to be definite (Ja jestem Czesław, a to jest Marysia), in the latter the object is usually indefinite (Ona jest studentką "She is a student), although not exclusively.
Cases. "chłopiec" is Nominative, "chłopcem" is Instrumental.
Nominative is mostly used for the subject of the sentence (Chłopiec je chleb), in "This is Y" sentences (To jest chłopiec) and it can be used in "X is Y" sentences (Adam to chłopiec).
Instrumental, apart from the meaning of "with sb/sth", is used in such sentences as here - "X is Y" (Ja jestem chłopcem).
Generally, chłopiec is younger and chłopak is older. However, the 'border' between them is totally subjective. One can say that it's 8 years old, another can say that it's 13... rather not later.
"chłopak" can also mean "boyfriend".
Frankly, "Jestem chłopakiem" on its own would sound strange to me. The sentence after all seems like something a child would say. Still, that's technically correct and accepted.
It's not in the nominative - it's dziewczynką, which is the instrumental. The little hook on the a is called an ogonek. If memory serves (it's been a while) the lack of it is probably just marked as a typo on Duolingo, but "Jestem dziewczynka" is actually wrong.
(Disclaimer: Not a native speaker and it's a long time since I did the course here.)
Let's count. In lesson 2 there's "Jestem kobietą" and "Ja jestem kobietą", as well as "Jestem mężczyzną" and "Ja jestem mężczyzną".
In lesson 3 there's "Ja jestem chłopcem" and "Jestem chłopcem", as well as "Jestem dziewczynką." and "Jestem dziewczynką!" (differing by punctuation). Looks like a draw to me. We don't control which of them you receive, it is possible that you got more male ones, but that's just by chance.