This is where Uncle Junior's Dementia started to come into play and ultimately led to the shooting of Tony in Season 6. I took it as Junior representing the last of the "Old School" Mobsters, and like all things, will fade into oblivion. Dementia patients can remember thoughts and events from 50+ years ago with absolute clarity, but cannot remember the last 10 words you spoke to them. Junior's only constant memory of Tony seemed to stem from a disappointment.

As for the "Varsity Athlete" comment and why it cut so deep, it was the idea of coming up short and being a disappointment/failure and not "special". With Junior and his HS Coach, they had high hopes for Tony and he let them down in his eyes. Now he has a son, AJ who could have been a varsity athlete, but had serious panic attacks, thus ending any chance at ball.

2 generations of "failures" would be pretty heavy for him to carry around like that.