"Földszint" is always the ground level. That is also its literal meaning.
If you are in a tall building and go down, say, from the 6th to the 5th floor, you say "lemegyek az ötödikre" (to the fifth). Or just "lemegyek X emeletet" (I go down X number of floors).