Is dobrou spelled with ou because noc is masculine?
No, noc is feminine.
Dobrou is an accusative form of dobrá. Accusative case is used mainly for objects of a verb.
Look at it as a shortening of Přeji ti dobrou noc. 'I wish you a good night.'
The verb přát něco 'wish sth' is used with accusative.