Nope, Czech works differently. "Do Polska" implies direction and cannot be used with "byli", as that implies location.
"Do Polska jsme byli pětkrát" is simply nonsense. You can however substitute "byli" (být) with a verb of movement, eg. "jeli" (jet) or "letěli" (letět).