By Dean Tudor
Journalism Professor Emeritus Ryerson University
Lately, there has been some researcher press about Deep Webs. Now,
these are not your Deep Throat type of sources. Rather, they are
part of the Web that does not turn up in search engines like Google
although together the major search engines can crawl through almost
one-third of these sites. Deep Web was once called "Invisible
Web" (coined by Dr. Jill Ellsworth in 1994), which I think
is still a better term. Deep Web actually means an accessible Web
site that not indexable or queryable by conventional search engines.
Invisible Web means not visible, and has wider coverage; it includes
both accessible Web sites (but not indexable or queryable) and non-accessible
Web sites (Intranets, passworded, non-coded).
Normally, indexed Web sites are displayed when you retrieve pages
from the various Web search engines. Subject directories such as
Yahoo will also display these sites. Some people call this the Surface
Web. The Deep Web (or Invisible Web) is what you cannot retrieve,
for you cannot see it in your search statements. Plus of course
all the URLs contained in these types of sites. There are seven
main divisions here:
1) The HTTP command is but a subset of Internet content. There
are also FTP (file transfer), E-mail, news, Telnet, and pre-Web
Gopher), which are not searchable.
2) Excluded pages: not every Web site wants to be totally included
in search engine reports. In the HTML source code there is a provision
and procedure for turning away the spider search bots so that they
don't report that particular page. In other words, the URL has been
turned off by the code.
3) Databases: many Web sites have the contents of thousands of
specialized searchable databases that you can search via the Web.
You can get results from these databases, but only in answer to
your one specific query. You cannot access the whole database. In
fact, the database may not even be stored online. You only wake
it up when it is needed for a response. "It is easier and cheaper
to dynamically generate the answer page for each query than to store
all the possible pages containing all the possible answers to all
the possible queries people could make to the database" (Berkeley,
see below). And thus the search devices cannot find or create these
pages. Tabular formats are a bitch to display without appropriate
software to do so. Even simple layouts such as crossword puzzles
have some display component, and hence are normally not indexed
(check out www.ecrostic.com).
Databases with tables created by Access, Oracle, SQL Server and
DB2 are accessible only by query. There's a lot of information out
there on the Web via databases. Content on the Deep Web may be 500
times larger than the normal Google-searchable Web. The 60 largest
Deep Web sources contain 84 billion pages of content. That's about
750 terabytes of information. Top dogs are the US National Climatic
Data Center, 366,000 GigaB, 42 billion records; US NASA EOSDIS,
219,000 GigaB, 25 billion records; and US National Oceanographic
Data Center, a mere 33,000 GigaB, 4 billion records. By contrast,
Google indexes only about 6 billion pages. And 95% of the Deep Web
is publicly-accessible information, not subject to fees or subscriptions.
4) For a variety of technical reasons (easy to understand, but
long and cumbersome to explain), there are "static" pages
on the Web. These reside on servers, waiting to be retrieved when
their URL is used in an HTTP command. But they are not linked, and
hence spiders do not find them.
5) Some sites require a password or loginID, and these sites are
closed to spiders. Passworded sites include indexing services, encyclopedias,
directories, Lexis and Nexis. In fact, any site that is not free
requires a password. There are thousands of such sites, although
some will let you in with teasers or partials, such as the Wall
Street Journal. Yahoo in 2005 made a small part of the
Deep Web searchable by creating Yahoo Subscriptions which searches
through a few subscription-only Web sites.
6) Non-HTML formatted pages: these include programming languages
which have codes that are incompatible with HTML, although the links
can be indexed but not the actual pages. Search engines have a hard
time with Adobe .pdf files (although Google has a reformatting tool),
image databases, spreadsheets (.xls files), multimedia files, PostScript
(.ps), Flash, Shockwave, PowerPoint (.ppt), and even wordprocessing
programs (Word .doc, WordPerfect .wp). There is no problem downloading
these materials once they are found; the major trick is finding
them in the first place!
7) Script-based pages with a ? (question mark) in their URL: these
are particularly devilish for spiders to locate. Most spiders do
not return the URL because of script problems and, believe it or
not, spider traps.
The basic Invisible Web, of course, are the various Intranets put
up by businesses, governments, and universities. These are locally
connected Web sites meant for just the corporation's use: sometimes
passwords are required. All manner of documents, many unclassified,
are posted - terabytes of information. And they are a major concern
for internal security since they can be hacked and also accessed
by rogue employees. There is no outside index to these sites, since
they are just local. All are hidden behind firewalls. I cannot tell
you how many times people have told me that a particular document
is on a Web site - "just go over and follow the links"
- only to find out that it is on their Intranet and hence inaccessible
to me. Actually, I can tell you: about a score of times
Other major invisible content includes static online library catalogues,
hidden portions of major Web sites, schedules and maps, complete
databases, tables of statistics especially in spreadsheets, phone
books, people finders (lists of professionals), patents, laws, dictionaries,
Web store or Web-auction products, newspaper archives, many blogs,
multimedia and graphical files. The Invisible Web is the fastest
growing category of new information on the Internet.
Also, "dynamically changing new information" will be
part of the Invisible Web. This includes news, job postings, travel
data (airline flights, hotels, etc.), stock market postings.
How do you find Deep Web? One way is through academic search tools
such as Infomine,
Index, and AcademicInfo.
You could try Direct
Search at www.freepint.com/gary/direct.htm. There is
also www.profusion.com, and www.completeplanet.com,.
Another way is through your usual search engine. Just type in a
short subject term with the word "database" (e.g., biomedical
database). If the database includes the word "database",
then bingo! (Bob's your uncle?). If you drill through a directory
such as Yahoo, then be sure to also use the term "database":
this will pick up additional listings. Many search engines feature
searchable databases as part of their service. Google, for example,
has separate searches for audio-visual material, images, news, and
non-HTML formats. These are just one click away from the main HTML
Some interesting Deep Web sites include:
* AnimalSearch (animalsearch.net):
family-safe animal-related sites, search by group, type, and regions.
* Educator's Reference Desk (www.eduref.org):
contains 2000+ lesson plans, 3000+ links to value-added online education
information, and 200+ question archive. It also provides access
to the ERIC database -- the world's largest source of information
on education research & practice, including free, full-text
expert digest reports, and it also links you to the Gateway to Educational
Materials (GEM), which "provides quick and easy access to over
40,000 educational resources found on various US federal, state,
university, non-profit and commercial Internet sites."
* NatureServe Explorer (www.natureserve.org/explorer):
"information on more than 65,000 plants, animals, and ecosystems
of the United States and Canada. Explorer includes particularly
in-depth coverage for rare and endangered species."
* Nuclear Explosions Database (www.ga.gov.au/oracle/nukexp_form.jsp):
Geoscience Australia's database provides location, time, & size
of explosions worldwide since 1945. Click on "databases".
* On-Line Encyclopedia of Integer Sequences (www.research.att.com/~njas/sequences):
"Type in a series of numbers and this database will complete
the sequence and provide the sequence name, along with its mathematical
formula, structure, references, and links."
* PubMed (www.ncbi.nlm.nih.gov/entrez/query.fcgiwww.ncbi.nlm.nih.gov/entrez/query.fcgi):
access to 16 million+ MEDLINE citations, including links to full
text articles & related resources. PubMed Central (PMC) is an
e-archive of free, full text articles from 200+ life sciences journals,
as well as Bookshelf, "a growing collection of [full text]
biomedical books (50+) that can be searched directly." Plus
the global NCBI 'Entrez' search engine for their many life sciences
* FindArticles (www.findarticles.com):
now searching 10 million+ articles from "leading academic,
industry and general interest publications."
* MagPortal.com (magportal.com):
freely available magazine articles on the Web, using keyword searching
or category browsing methods.
* Directory of Open Access Journals (www.doaj.org):
one stop open access directory, providing no-cost access to the
full text of over 2,000 journals, with over 500 journals searchable
on the article level (over 83,000 articles available) in the science
and humanities/social sciences
* Cryptome (cryptome.org):
specializes in posting both previously classified or under-publicized
US federal documents, along with similar documents from other jurisdictions.
There could be half-a-dozen posted every business day. Just go over
to the site, and the home page lists the latest docs. Typical titles
include "Expansion of the Strategic Petroleum Reserve",
"Calendar of 2,482 US Military dead in Iraqi War", "Security
Measures for Radioactive Materials", "Outer Continental
Shelf Polluters Fined", "CIA Creation Documents".
There is also an index to off-site documents, dealing with topics
such as the Israeli Lobby and US foreign policy, Al Qaeda documents,
New York City public safety.
Cryptome is a true Web site, with multiple links to other similar
document retrieval efforts. You could do worse than beginning with
Cryptome for searches involving nefarious actions of government.
The site also has a searchable data DVD of its archives of over
33,000 files (since June 1996), just under 3 GB worth.
There is actually a firm promising to locate Deep Web material.
It is BrightPlanet (www.brightplanet.com).
Their mission statement: "BrightPlanet applies unique and fully
automated technology to internet search site identification, retrieval,
qualification, classification, summarization, characterization and
publication". Currently, BrightPlanet software is configured
to query 70,000+ Deep Web sources. It'll even walk your dog
For the immediate future, you should expect a big impact from two
One is the court system. The
Canadian Judicial Council (the organization of Canada's
top judges) has recommended that access to court records via the
Internet be restricted. Many of these may be moving over to the
local intranet and never accessible via the open Internet. You'll
soon have to visit your nearest courthouse to view legal documents,
much as you have to now just to view the paper copies.
Remote access would still be available for judicial decisions and
case information, but not to affidavits, motion records, and pleadings.
All in the interests of privacy and identify theft. It is one thing
to have publicly available documents at the courthouse (you must
first determine which courthouse and you must ask for the right
piece of paper), but it is quite another thing to have publicly
available documents floating out on the Internet where just anybody
can read them. Yes, they are public, but only in paper form and
U.S. courts are still more open. Documents
from scores of federal courts can be downloaded through PACER (Public
Access to Court Electronic Records) for a small fee, by the page.
Another is change of ownership. While most of the databases within
the Deep Web are government-owned or non-profit, there are still
vast areas such as E-mail and FTP which are in private hands. Every
time someone buys an Internet property, there are policy changes.
What should we expect with the newest batch of dot com purchases
by the media itself? How will this play out for searching for data?
NBC Universal has bought iVillage,
the top women's oriented site on the Internet, with over 30 million
unique visitors a month. News Corp (Murdoch) has bought MySpace,
the fastest growing social networking site on the Web, about 50
million unique visitors a month. News Corp also bought IGN,
a top gaming and entertainment site for young hot males, with under
20 million unique users a month. Viacom (owner of MTV and Paramount)
has bought Neopets,
a young person's community site with virtual pets. Viacom also has
(where users track the film industry and post their own videos),
(a competitor to IGN with more hot males), and GoCityKids
(via Nickelodeon). The New
York Times has bought About.Com,
an online advice site with over 60 million unique users a month.
Other hot properties appear to be photo- and video-sharing sites.
Murdoch still has $2 billion earmarked for these purchases, coming
up real soon. The big audiences in all the new acquisitions can
link to each other within and without their communities. And they
could be susceptible to database searching by new owners or positioned
for a sell off of contents to database searchers.
For more details on the Invisible Web and the Deep Web, try these
Published in Sources
58, Summer 2006.
Dean's Digital World Articles
Sources, 489 College
Street, Suite 201, Toronto, ON M6G 1L9.
Phone: (416) 964-7799 FAX: (416) 964-8763
Include yourself in Sources
Mailing Lists and
Media Names & Numbers
Names & Numbers