In English, any vowel can sound differently depending where it is in the word. Take "Ball" versus "Cat." Same rules apply to Russian; if it isn't stressed, it often changes.
Yep, and they're both in this sentence. Хочу uses the stressed "oo" and собаку uses the unstressed "oh." Also of note would be the letters O in the sentence; they are unstressed as well, and so sound like an A. If you want a full description in depth, look up Russian Vowel Reduction; it might seem like there are a lot of rules to it, but the central theme is that unstressed vowels require less effort and articulation than stressed do.
If a male cat is Кот while female cat is Кошки, then is there a male equivalent for собака?
male собака - КОБЕЛЬ (ударение на 2-м слоге), female собака - СУКА, СУЧКА (ударение на 1-м слоге). All these are widely used as swearwords.
It is собаку because it is the direct object of the sentence. It is the thing the speaker wants. The direct object is in the accusative case, so собака (nominative case for 'dog') changes to собаку (accusative case for 'dog'). I hope this helps.
Would "I want dog" be correct? (Say you are in China and would be eating it at a restaurant, for example.)
If you were in China, you'd probably not say it in Russian. Rather, you'd say: Wǒ xiǎng yào yī zhī gǒu