They used to beat up New York wannabes there.

If I had to pick only ONE neighborhood as the toughest I'd go with either East New York OR Brownsville, (Brooklyn). Both neighborhoods could argue for that title (of toughest) over a period of generations (Brownsville was Jewish in the '20s/'30s/'40s and home to Murder, Inc; later became black and home to street toughs like Floyd Paterson and Mike Tyson) ~ (East New York was heavily Italian in the early periods listed above - remember the early parts of "Goodfellas" took place there - and later Hispanic and black when city housing was built in the '50s).