What Makes a Piece of Land So Much More Valuable?

An acre in the middle of downtown Manhattan is ruinously expensive, even if there's absolutely nothing on it. The tricky part about this then is that if you have a piece of land and it goes up in value without you doing anything to it, it's because the people around you were doing things. So you have profited off the labor of everybody else around you without you having to do anything at all.