He was a former police officer. That information isn't discussed in the film, but they tell a good deal of Neri's backstory in the novel. Basically, he was called to a crackhouse, where he subsequently killed a drug dealer who was raping a woman. He got in trouble with the department, Michael Corleone recruited him, and the rest is history (in a nutshell, I'm sure JustMe, Turnbull or some of the others can give a more accurate re-telling of Neri's story).