Your second sentence could also be written as 'Waarom erger jij je aan kinderen?', with the jij being the subject pronoun (equivalent in your first sentence: ik) and the je being the reflexive pronoun (equivalent: me).
How is this sentence different from " I am annoyed "at" my brother. In English there is a difference here with the use of "by" vs "at". In other words are we saying here that I am annoyed at my brother ?