It does. Names of languages and the related adjectives are capitalised, just as in English:

• ysgol Gymraeg - Welsh-language school (soft mutation of Cymraeg -> Gymraeg following the feminine noun ysgol).
• Llenyddiaeth Almaeneg - German-language literature
• Roeddwn i'n mwynhau dysgu Lladin a Ffrangeg yn yr ysgol - I used to enjoy learning Latin and French at school.

(Note that 'Welsh', as opposed to 'Welsh-language' is Cymreig.)