It's a known fact that John feared the Roy DeMeo crew. I think he was also intimidated by Sammy. Not in terms of power or strength but by how prodigal an earner Sammy was. I think a part of him was always afraid Sammy would one day try to step over him to be Boss as he did with Paul.