"What was she saying?"
First I thought it was turning 何 into an adverb but I couldn't see the use of the adverb here, so I looked around a bit.
Take a look at this: http://www.guidetojapanese.org/learn/grammar/actionclause
But it basically works as a quote, you wanna know what -exactly- she was saying. Can be used with saying, hearing, thinking and such.
With と doesn’t it mean “Was she saying ‘nani’?”
Because と is a quoting particle.
Whereas with を it’s means “What was she saying?”
As in what was the object of what she was saying.
.....if that makes any sense.
It is completely fine to drop the い in the ～ている form and yet this question marks it incorrect.
言いました (iimashita) is past tense- "said" "what did she say" used when an action happened and finished. Emphasizes the action of having spoken.
This sentence takes past progressive 言って いました (itteimashita) "was saying" "what was she saying" used when an action was ongoing, puts more stress on the duration and other actions that were taking place at the same time. "What was she saying (when X was happening?)" Or "What was she saying- (before being interrupted)"