So I googled a bit for americans with no passport...and most websites mention that only about 30% own a passport. That's not official of course. Neither is the following bit from wikipedia, which argues for a rise in number since 2006 :

"For many years, passports were not required for US citizens to re-enter from countries near the United States (including Canada, Mexico, Bermuda, and most Caribbean and Central American nations.) In light of this, and given the country's immense size and the great distances which the average citizen lives from an international border, passport possession in the United States had remained relatively low. Indeed, most Americans normally did not obtain passports or carry them regularly unless traveling abroad, and as of 2006, only 60 million (20% of Americans) had passports.[7] As of 2011, approximately 37% of Americans have Passports or Passport cards"

http://en.wikipedia.org/wiki/Identity_documents_in_the_United_States