Shouldn't “se” be the second word in the sentence? Is “Matěj by” considered an inseparable unit?
The second position rule is used for all clitics, in general. Obviously, you can't have them all in one position so you need to arrange them accordingly. The conditional auxilliary forms ('by, bych...') take precedence over 'se'.
shouldn't it be "matej by ses nebal niceho?"
"Ses" is actually used for second person singular :-)
"ses" in this sentence is incorrect.
As far as I understand, both 'níc' and 'ničeho' mean "nothing". In what cases should we prefer one over another?
Isn't “ničeho” the genitive of “nic?” I think “bát se” simply requires the genitive, so one cannot use “nic” (nominative) here. In a similar way English requires the “of” after “afraid.”
It follows all the same rules as all other words. Bát se requires genitive for the direct object. It does not matter what part of speech (noun, pronoun,...) it is, as long as it has cases.