Can you also say małą
That's very unncommon and sounds awkward to some, better not to use it. The adjective should agree with noun in its grammatical case..
Is it instr. case (małą)?
yes małą is instrumental singular feminine.
You may say it, you might hear Poles say it, it is not correct, unless before a noun.
Thank you :)
With małą the sentence:
Tamta ryba jest małą rybą
How come it's 'that' and not 'this fish'?
Well, first you can check what immery usually writes to questions about this/that: here
So we usually accept both 'this' and 'that' for 'ta' (at least we try), but if it's 'tamta' in Polish, it really needs to be 'that' in English.
Do adjectives match with their nouns in their grammatcal case even after verb "być"?
Yes. Adjectives match nouns. If there is no noun after być the adjective is in nominative.
Why no instrumental case in such sentences?
You use Instrumental in such sentences for noun phrases, but not if you only have an adjective. Maybe it will be easier to show with a masculine noun:
"Lew jest duży" (Nominative)
"Lew jest dużym zwierzęciem" (Instrumental)