Shva is either pronounced as e or not pronounced at all. It's marked like you wrote, because it's pronounced "yom revi'i", but in certain instances, shva becomes silent. And marking it with the special letter helps us realize when "e" sound is segol or tsere, an actual e sound, and when it is shva.
His question was, how did we get from ארבע to רביעי when initially they look very different, compared to, for example חמש and חמישי.
There are many instances where initial א ia not part of the root, but a prefix added for some reason. That is also the case here. So, you take the א and you're left with רבע which is the root for things quadrilateral, or quarters and so one - in other words things that have to do with 4. So, רבע becomes רביעי - fourth.