Why is it "byly" and not "byli", when "míč" is masculine?
It is masculine inanimate. Only masculine animate have "byli".
what about "Byly na hřišti dva míče."?
As an introduction to a fairy tale perhaps, otherwise no, that is a question word order.
And what about "byly dva míče na hříšti"
The same. It is not impossible, but normally it is very unnatural unless it is the first sentence of a story (typically a fairy tale) you are telling.