Now that I look at it, the question is also about yonpun vs yonfun.
Regarding yoji, I searched around, and here's what I've found. One of the ways to pronounce the kanji is よ (yo):
Correct, four can be pronounced as 'yon', 'yo' or shi. But when used for hours it is always 'yoji'. Nine o'clock is also 'kuji' instead of 'kyuji'.
I am a native English speaker from NZ, living in Australia. We commonly say how many minutes 'past' the hour until you reach the half hour. Then it is common to say how many minutes 'to' the hour for times after the half hour. E.g. 4:15 we'd say "quarter past four", 4:30 we'd say "half past four", and 4:45 we would say "quarter to five".