Because of damn! It makes me think of cursing that dust that always comes back, or I suppose it could also refer to the low value coin, the dam where we got the phrase, "It's not worth a tinker's dam." Dust is not very valuable! :) I love mnemonics.
Tinker's dam was new to me. But, do you know that "damm" also means dam, not the tinker thing, but the water barrier? And "dam" (long Swedish a) means lady :).
Thanks for that clarification. My first thought actually was "my brother's house is full of women" :-)