Kolik bylo mrtvých tam? Why is it wrong?
The 'there' in the question isn't referring to a place, but a state of past existence. 'There were XXX dead' as opposed to 'There (location), there were XXX dead.' So, the 'tam' in your answer isn't necessary.
But the way the question is written,'there' could be referring to a place. It could equally mean 'How many were dead?' or 'How many dead were in that place?'
There is not way to know if there is place or "at that time". I also thought if was referring to some place
I wrote: Kolik mrtvých tam bylo? and it was accepted.
It is wrong word order. "Kolik tam bylo mrtvých?" or "Kolik mrtvých tam bylo?" is OK.
Is mrtvých a noun or an adjective in the genitive here? If a noun, what is the nominative form for this?
It is an adjective meaning "dead". The masculine nominative is mrtvý.