Akkusativ oder Dativ?
I am saying “Tom is called my other brother “ - Tom heißt mein anderer Bruder OR Tom heißt meinen anderer Bruder??
Yes, it is the nominative case.
I doubt that the method of asking for a noun (wer oder was / wessen / wem / wen oder was) helps someone who isn't a native German speaker to determine the correct case. That is basicly a test for what sounds right to us or how we understand the world.
Tom heißt mein anderer Bruder.
English is not my native tongue so I am not sure about this, but the word order of the English sentence seems odd. I'd take the English sentence talking about a Tom who ist not really my brother but is called so. If that is the meaning of the sentence, I'd use "Tom wird mein anderer Bruder genannt." or "Tom wird als mein anderer Bruder bezeichnet."
No, it's fine.
From the word order, I would assume that the context is something like, "Oh, you're one of Stepintime's brother's, aren't you? Tom, right?", to which I reply, "No, this is my brother Tim. Tom heißt mein anderer Bruder." (Tom is the name of my other brother, not of this one.) - or - "I'd like to invite you, your brother Tim and your other brother to my birthday party." "Tom heißt mein anderer Bruder." (The name of my other brother, which you can't remember, is Tom.) Generally, you might think of it as "Tom is what my other brother is called."
If you just want to say that you have another brother named Tom, you'd say, "(I have two brothers. One of them is Tim, he's a baker and lives in Berlin.) Mein anderer Bruder heißt Tom."