If Jimmy's Last Mafioso book is anything to go by, I think Warren had some sort of illness which kept him from really getting involved in that life along with his brother. I think Jimmy makes countless references to having to "take care" of his brother or "looking after" his brother in that book. I'll have to go look again as I'm not totally positive. But I always got that implication from the parts of the book where he's mentioned.