In my head, the primary meaning of "někdy" is "sometimes," but that doesn't make as much sense here as "ever" does, if you think about the meaning of the sentence, from the speaker's point of view. The problem is, we don't really know what that point of view is until we get this wrong and find out that we should have used "ever."
No way around this, I guess, other than "That's just the way it works"?
I have added "sometimes.” It’s heavily reported, not grammatically incorrect, and I can imagine it being used with an emphasis on the word. But I would add that "ever" is a much better choice in terms of the way Person A would ask this question when s/he is not pleased that Person B seems to be regularly disregarding what “A” says. (I have also added "Do you listen to me ever" because I can easily imagine that being used these days, especially in the US, with an emphasis on "ever").