I think there is a relation between working and jerk/scumbag. The conquered populations were the main source of labour force for the Klingon empire. "They are more than slaves, but less than citizens", according to scifi.stackexchange.com/. So, we can figure that "the workers" are assimilate with "conquered people", which lost their honour, being defeated. That is just my humble opinion.