In fact, "que" becomes the direct object pronoun which replaces "la lettre".
Consider, C'est la lettre. Nous avons écrit la lettre. No agreement here.
Rather than mentioning la lettre twice, the direct object noun, "la lettre" can be replaced with the direct object pronoun " l' " to make;
C'est la lettre. Nous l'avons écrite.
Agreement is with " l' " which is feminine and not with the subject pronoun, "Nous" as it's using an avoir auxiliary verb rather than être.
C'est la lettre que nous avons écrite.
Combining the sentences, "que", the relative pronoun is used to replace " l' " which is feminine and hence the agreement and it also occurs before the verb.
If I understand correctly, it's because "la lettre" is mentioned before the verb, rather than after. When it's mentioned before (la lettre que nous avons écrite), then the conjugated verb has to match the noun - in this case, feminine and singular. If it's mentioned after (nous avons écrit la lettre), then there is no agreement. At least I think that's it...
Technically speaking the distinction cannot be made unless there is a temporal adverb. "It is the letter we wrote" suggests the letter was written at a specific, known time in the past ("yesterday," for example). "It is the letter we have written." suggests the letter was written at some general, unknown time in the past ("before," for example). Most of the examples for this section do not come with such adverbs, and so the present perfect and the simple past should be interchangeable.
The second that is a relative pronoun and can be omitted as you did in your 2nd comment. However my explanation above is related to the first that which is before letter...
By the way if I want to back-translate what you said into French, it would be sth like:
C'est cette lettre-là que nous avons écrite. which is different than what this question is.
We wrote that letter = Nous avons écrit cette lettre = On a écrit cette lettre (more informal than the previous)
If these two sentences are written in two different forms (and the structure is clearly different), why should French be an exception to that? It may have its own equivalents to each of them...