Newsletters

Select newsletters below and click the button to sign up!

Boston News NY News
DC News Internet Daily
SiliconValley News
InternetNews Business Report




Become a Marketplace Partner



Partner With Us















Internetnews Bloggers

Recent Entries

Archives

April 2009
Sun Mon Tue Wed Thu Fri Sat
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30    

Monthly Archives

Search The Blog

Netstat -vat by Sean Michael Kerner (bio)

A command line view of IT



1 trillion unique URLs on the Web

google.logo.jpg
From the "Carl Sagan would be proud" files:

Google is now reporting that its seen more than 1 trillion unique URLs on the Web. That's a big number and a massive jump from the 26 million URLs it saw in 1998.  But even at a trillion pages Google admits that there are many duplicates (in terms of content), and that there well may be an infinite number of pages overall.
Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual Web pages out there is growing by several billion pages per day.

So how many unique pages does the web really contain? We don't know; we don't have time to look at them all! :-) Strictly speaking, the number of pages out there is infinite
.
Thanks to the magic of dynamically generated page content (with or without session IDs) and the fact that Google (despite their best efforts) has never effectively indexed Flash content properly -- I personally think the 1 trillion number is on the low side.

Now that doesn't mean there are more than a trillion Web sites out there -- the latest netcraft study reports just over 173 million sites.

| Comments (0) | TrackBacks (0) | Share

0 TrackBacks

Listed below are links to blogs that reference this entry: 1 trillion unique URLs on the Web.

TrackBack URL for this entry: https://swarm.jupitermedia.com/mt-tb.cgi/4129

Leave a comment