Why od instead of niż?
you can use both interchangeably , but od +genitive, niż + nominative
This should also work (though I haven't tried yet)
Twój pies jest mniejszy niż mój.
That is correct.
why not 'niż mnie'?
That means "than me", and also it would only work in some grammatical context which need either Accusative or Genitive, for example "Ona lubi ich bardziej niż mnie" (She likes them more than me, She liks them more than she likes me).
Kocham cię bardziej niż ty lubisz mnie
https://youtu.be/8lUvr4sDlQA