In Manhattan, going “downtown” means going toward the southern tip of the island, traveling in the direction where street numbers are lower. In most other cities, going “downtown” means going to the central business district — basically located in midtown. Except in addition to the midtown downtown, there’s the other downtown actually located all the way downtown by the Mosque Exclusion Zone. I’d always heard that the reason for these two separate clusters of skyscrapers is that the skyscraper-free neighborhoods in between — where I grew up — are located on top of a portion of the island where the bedrock isn’t suitable for skyscraper construction.
According to Jason Barr and Troy Tasier’s paper (PDF) “Bedrock Depth and the Formation of the ManhattanSkyline, 1890–1915” this is basically an old wives tale:
Skyscrapers in Manhattan need to be anchored to bedrock to prevent (possibly uneven) settling. This can potentially increase construction costs if the bedrock lies deep below the surface. The conventional wisdom holds that Manhattan developed two business centers — downtown and midtown — because the depth to the bedrock is close to the surface in these locations, with a bedrock “valley” in between. We measure the eﬀects of building costs associated with bedrock depths, relative to other important economic variables in the location of early Manhattan skyscrapers (1890–1915). We ﬁnd that bedrock depths had very little inﬂuence on the skyline; rather its polycentric development was due to residential and manufacturing patterns, and public transportation hubs.