Yes, खिड़की is feminine, so in plural form it is खिड़कियाँ. For feminine plural nouns the adjectives conjugate with ी (so नयी here). On the other hand दरवाज़ा is masculine, so दरवाज़े is the plural, and masculine plural nouns conjugate with े (नये here).