The rule is rather simple: the vowel depends on the consonant preceding the -ка. The vowel is usually unstressed, and in this case the following rule works:
- О after a non-palatalised consonant (e.g. кнопка→кнопок)
- Е after a palatalised consonant (дочка→дочек)
- treat Ш, Щ, Ч, Ж, Ц as palatalised, at least in spelling. Pronunciation-wise ш, ж, ц are actually non-palatalised.
If the vowel is stressed, it will always be О, even after Ш and Ж. I find it hard to provide a relevant example apart from кишка→кишок. I flipped through the dictionary and could not find any other nouns ending in -жка, -шка or -цка that would have the word final stress.
My understanding is that the software permits you to have one incorrect letter per word. Exception being if the incorrect letter makes a different word, in which case it assumes you are incorrectly writing a different word. Therefore ощибок must be a different word (?). either that or you also got something else wrong.