Nope, Czech works differently. "Do Polska" implies direction and cannot be used with "byli", as that implies location.
"Do Polska jsme byli pětkrát" is simply nonsense. You can however substitute "byli" (být) with a verb of movement, eg. "jeli" (jet) or "letěli" (letět).
The word order is a little "non-standard" but it is not grammatically incorrect. After consultation, "We have been five times to Poland" has been added as an alternative translation.