If there was no wind, then it would land back at your feet, due to conservation of momentum. The air mass is rotating with the Earth, and can be considered as one. If the ball goes high enough to escape the bounds of this system, then it will indeed be somewhere else when it re-enters the system, assuming as you say that nothing interferes with the straight up and straight down path. Time and altitude has nothing to do with it as such; it's all dependent on leaving the Earth system.
Now, assuming it doesn't leave the system, this is going to surprise you (probably).
As the ball rises, it will appear to move to the west, as you might expect. However, as it falls, it's angular velocity is accelerated by the Earth's gravity, and it actually moves back to the east. The ball falls at your feet... There have been demonstrations of this effect by dropping balls down very deep mine shafts. It's difficult to perform, as the slightest air movement skews the results, but for a shaft that's >100m deep, the effect is a couple of cm or so, but it's to the SOUTHEAST of where it started, not to the WEST as you might imagine. The deviation to the south is due to the orbital fall of the ball following a great circle, not the line of latitude.
Helpful?