why is this wrong? você ama a minha irmã?
It's accepted without any change in meaning.
Would it be possible to use the verb 'querer' instead of 'amar', like Spanish?
No, you couldn't. Despite the verbs "querer" and "amar" being synonyms in Spanish, in Portuguese "querer" means "to want".
Why did reject "você amam minha irmã"?
Você ama (singular)
Vocês amam (plural)