"There is a big library near my house."
Nope. Placing 図書館が at the beginning of the sentence suggests 'The library (which has been mentioned) is near my house.', rather than 'Near my house, there is a library.', or the version which DL provided.
The focus of the sentence changes from informing as to what is near my house to where the library is.
It is a possessive particle. If you translate 近く (noun) as 'vicinity', the sentence would be akin to 'The house's vicinity' or 'The vicinity of the house'.
Using 'near' (an adverb) instead of 'in the vicinity' (a noun), while more colloquially common in English, is farther from the grammar of the Japanese sentence.
There is some inaccuracy here. 大きい and 大きな are 2 separate words although both of them means big. 大きい is not a Na-adjective and it changes form according to I-adjective rules.
The difference in usage is 大きな is only used as an attributive adjective before a noun whilst 大きい is used also in other functions, and changes accordingly.
Which one to use before a noun is a matter of Japanese usage habit. Usually 大きな is used on conceptual, intangible matters while 大きい is used on physical things. It seems though a lot of overlapping exist and using either one should not be considered wrong. So yes, 大きい can also be used.
Not sure that's the best explanation - it's not inconceivable that a discussion might have been about various types of library, and you wanted to note "As for big libraries, there's one near my house", in which case は would be appropriate (though it would be unusual to put it so late in the sentence), especially if you were contrasting it with other buildings or types of libraries. But in the normal scenario that you're simply pointing out the information that a big library exists near your house (think of it as a response to a question, implied or otherwise, such as "what's in your neighbourhood"), then yes, が is appropriate. But Duo is very fussy, and many exercises simply require you to guess what context the contributors had in mind. In some cases there's little logic to it (e.g. for "A dog bit my leg", they only accept 犬は).
そば generally signifies less distance than 近く. As distance is relative and there will often be some overlap そば might be substituted for 近く in some cases. It depends on whether the speaker considers A to be near B or very near B, which is subjective. For this sentence in particular, it does not appear (to me) that the library is close enough to use the word そば.
そば is close enough that it is often means 'right by' someone, or figuratively 'within arms length'. It is often used to show intimacy and there can be an emotional component to this word. As an example, 君のそばにいたい is often translated as 'I want to be with you' or some similar paraphrase.