Doctor Foster went to Gloucester in a shower of rain...., as the nursery rhyme goes. It rhymes with "Foster".
The first syllable rhymes with "gloss, boss, toss"; the second is an unstressed syllable as in "faster, better, mister".
I'd pronounce it the way kdb119 wrote in IPA -- I distinguish the vowels in "cot - caught" and in "bother - father". If you're from the US, you may not have this particular version of "short O". And the second syllable has no "R" sound in it for me -- my accent is non-rhotic. (See also https://en.wikipedia.org/wiki/Rhoticity_in_English .)
I'm not sure what the traditional pronunciation in Gloucester itself is; it's far enough to the south and west that the traditional pronunciation there may actually be rhotic (as are most varieties of English in the US).
Hmmm, a little tricky to describe without IPA, the OED suggests /ˈglɒstə/ but my best attempt would be 'Gloh-ster'.
I'm trying to think how not to imply 'Glow-ster' which would be wrong - although I suppose some West-country accents might approach that pronunciation.
Perhaps someone with a better appreciation of how to convey pronunciation might have a better stab at this?