"I only got off work at 11 last night."
才 (CAI2) should be used when an action occurs later than expected. For example, 我昨天晚上十二点才回家 (I didn't return home until 12:00 last night).
The opposite of 才 is 就 (JIU4) and should be used when the action occurs earlier than expected. For example, 我昨天晚上七点半就回家 (I already returned home by 7:30 last night).
Here is a good video https://www.youtube.com/watch?v=ebwgMkQ3jMo It says cai is best thought of as "only just" and has an implicit meaning of "...didn't do until..." For that reason, in this sentence, it is more accurate than using Zhi, which is more for a limitation (eg I only have three pens")