Swedish does have a singular - peng. It's not very common and it would usually be used either to mean "coin", or "a small amount of money".
For instance: vi gav dem en liten peng = we gave them a little money.
It's also used in compounds, e.g. a child's weekly allowance = veckopeng.