Just to make sure, in this case does szklanka only means glass, the drinking vessel, not glass the material?
Both of those seem right, but "kubek" is definitely "cup" too. It even translates as a unit of measure.
What is the term for just a regular cup--like a reusable "glass" that's made out of plastic?
Szklanka is derived from szkło (glass), so everything out of plastic or china should be kubek. Although, a glass with a handle could also be kubek (szklany), and a transparent plastic cup without a handle (that looks like glass) can be szklanka (plastikowa).
Ok, now I have to admit that while writing this, I got confused myself :)
If you want to emphasize the fact that it's made out of plastic, you can say kubek plastikowy. If it's really important to point out that this cup is also reusable as opposed to disposable (jednorazowy), you can add wielokrotnego użytku, but I've actually never heard anyone say that out loud.