You'll find useful info on this subject in Harlan Lebo's "The Godfather Legacy" (a must for any serious player on these boards). Briefly: only Duvall had been around long enough to establish a tangible reputation. FFC chose him and Caan because he'd worked with them previously on "The Rain People." Keaton had done theater, TV commercials and was a member of Johnny Carson's "stock company." Pacino was up-and-coming, but still relatively unknown.
One thing that's not said explicitly, but which I suspect: FFC may have chosen them in part because they were relatively unknown, and he could therefore get them cheap. One of the reasons Paramount chose FFC to direct GF was that he was young and relatively unknown, and they thought he might be able to bring GF in cheap(!!!) He wanted Brando and knew he'd have to pay big bucks for him, so perhaps he chose the others in part because they didn't command high salaries. As you probably know, Pacino got all of $50k for his work in GF.