I have a theory...

He claimed he was a retired construction worker in his later years which leads me to believe that he was masquerading around as a construction worker from the 1970's onwards.
He would have been forced to continue with this ruse which would have molded his modest lifestyle to match up with his claim of being a construction worker/retired construction worker.

Just like 'Chin' he could never live it up and enjoy life because he had to "keep up appearances".