Its a good question, but in a sense its asking to compare apples and oranges. Don Vito would NEVER have killed a brother. Hell, he couldn't even bring himself to kill a son-in-law, fully knowing that the son-in-law was responsible for the death of his own son.

Part of the effect of Michael killing Fredo was to show how "Americanized" (and thus, less soulful) Michael had become. Vito had "lost his family" through no fault of his own; Michael couldn't make that same claim. The times had changed, and Vito was a throwback to some older times that no longer existed.

I think Vito would have simply banished his brother to an area from which the brother couldn't have been a threat.