is "my mother is writing a letter" not right? does the "je" change brief to note?
Yes, the diminutive implies it's something small/quick, so e.g. a note or a shopping list. Een brief will also never be a note, but always a letter.
Can "briefje" be letter? or small letter?
I don't think so, I would always use korte brief for a small letter.