Remember being a small child and all the places adults would drag you to would seem so... huge?
Then you grow up and become an adult and those places feel so much smaller, way less intimidating.
When you take away all the people in those places (a grocery store, a mall, a school) they suddenly become very large again. All those anxieties or imagined fears from childhood can easily come flooding back.