So how are we supposed to infer that it the subject is he/she? Why could it not be I or they or you since there is no subject prefix?
The absence of the prefix does not mean the subject could be anything, it specifically means that it is something other than you or me. "I", "we", and "you" must be represented by prefixes. The null prefix (that is, the absent prefix) must represent a third person subject and an absent or third person object. Thus "he", "she", and "it" are all accepted in the translations of this sentence.
The question of "they" is a little more complicated. The one exception to the third person subject and object using the null prefix is when the subject is plural and the object is singular. This will be taught in an upcoming Skill, but in this case, there is an actual prefix: lu-. Thus, to say "They do not understand the man," you would have to say loD luyajbe'. But if they are both third person plural, then you would still use the null prefix.
As mentioned in the Tips & Notes for this Skill, the plural markers are optional. Thus, if you read loD as possibly meaning "men", then both the subject and object could be plural and "They do not understand the men," is also an accepted translation.