The lack of black people in the GF was a quite accurate reflection of pre-civil rights movement America. Don't forget that until the late 40s, even the Armed Forces were racially segregated. Until 1947 baseball was racially segregated. If you peruse the newsreels of the 40s taken at ballparks you won't see any black faces. When did football (pro or college) integrate?

You might also remember a line in the novel where Clemenza decides against promoting someone to take Paulie's place because he got along too well with black people which indicated a flaw in his character. Yes, both the novel and film accurately represent the racial divide in America at the time.