You can think like this:
Aqui = here (near the person who is talking - 1st person)
Aí = there (near the person with whom you're talking - 2nd person)
Lá = there (near the person or thing or place of which the first and the second people are talking - 3rd person)
Ali = there / over there (near the first and the second person - 1st person plural)
PS: "ali" neither too near, nor too far. It's a short distance.
If it has an "l", it's not nearby. Aqui and ai are near while ali and la are not. And la can be really far, like in la-la-land.
I like your point! And I'm going back to Florida soon. Aí onde você está