My dictionary gives me "horší" for "worse", and does not know "hůř". Can you explain the difference, and why horši is wrong here?
Horší is the adjective, hůř is the adverb. (Both are the comparative forms.) You need adverb in this sentence.
špatný - horší - nejhorší
špatně - hůř (hůře) - nejhůř (nejhůře)
What would "Psal ani hůř, než mluvil." mean, if anything?
That means nothing. "ani" is only used in negative sentences.