Deduplication and the Ephemeral Web

### All knowledge, to all people, for all time, for free Universal access to all knowledge, Kahle declared, will be one of humanity's greatest achievements. We are already well on the way. "We're building the Library of Alexandria, version 2. We can one-up the Greeks!" Start with what the ancient library had---books. The [Internet Library](http://www.archive.org/) already has 3 million books digitized. With its Scribe Book Scanner robots---29 of them around the world---they're churning out a thousand books a day digitized into every handy ebook format, including robot-audio for the blind and dyslexic. Even modern heavily copyrighted books are being made available for free as lending-library ebooks you can borrow from [physical libraries](http://www.archive.org/)\---100,000 such books so far. (Kahle announced that every citizen of California is now eligible to borrow online from the Oakland Library's "[ePort](http://oakland.lib.overdrive.com/B5827532-F62F-49BC-8C8E-8BF0DDBD600A/10/644/en/Default.htm).") As for music, Kahle noted that the 2-3 million records ever made are intensely litigated, so the Internet Archive offered music makers free unlimited storage of their works forever, and the music poured in. The [Archive audio collection](http://www.archive.org/details/etree) has 100,000 concerts so far (including all the Grateful Dead) and a million recordings, with three new bands every day uploading. Moving images. The 150,000 commercial movies ever made are tightly controlled, but 2 million other films are readily available and fascinating---600,000 of them are accessible in the [Archive](http://www.archive.org/details/movies) already. In the year 2000, without asking anyone's permission, the Internet Archive started recording 20 channels of TV all day, every day. When 9/11 happened, they were able to assemble an [online archive of TV news](http://www.archive.org/details/911) coverage all that week from around the world ("TV comes with a point of view!") and make it available just a month after the event on Oct. 11, 2001. The Web itself. When the Internet Archive began in 1996, there were just 30 million web pages. Now the [Wayback Machine](http://www.archive.org/web/web.php) copies every page of every website every two months and makes them time-searchable from its 6-petabyte database of 150 billion pages. It has 500,000 users a day making 6,000 queries a second. "What is the Library of Alexandria most famous for?" Kahle asked. "For burning! It's all gone!" To maintain digital archives, they have to be used and loved, with every byte migrated forward into new media every five years. For backup, the whole Internet Archive is mirrored at the new [Bibliotheca Alexandrina](http://www.bibalex.org/Home/Default_EN.aspx) in Egypt and in Amsterdam. ("So our earthquake zone archive is backed up in the turbulent Mideast and a flood zone. I won't sleep well until there are five or six backup sites.") Speaking of institutional longevity, Kahle noted during the Q & A that nonprofits demonstrably live much longer than businesses. It might be it's because they have softer edges, he surmised, or that they're free of the grow-or-die demands of commercial competition. Whatever the cause, they are proliferating.