Does it mean 'What is wrong with father?' or literally 'What [object] is [physically] with father?' ?
It can mean both, just like in English ("What's with dad?" - it depends on the context that you are asking what's wrong with him or what is he holding, for example).
It's sort of like saying, "what is wrong with father?" It's a bit of a colloquial phrase.
For non-native English speakers confused about our colloquialism, think about it like this. "What is with father?" colloquially means "Whats wrong with father?"
Consider this, "What (trouble) is with father?" It's as if you are asking what trouble is with him. You know father is troubled but you don't know what is troubling him, so you're asking what "trouble" is with him, but you don't actually say trouble, hence- "What () is with father?"