In the 70s and 80s ordinary people could actually afford to live in Manhattan. What people forget is that the reason the city was affordable is that it had a high crime rate.
There were many neighborhoods that were so drug and crime infested that people were scared to go there.