The Invisible Web: What It Is, How You Can Find It

The Invisible web is out there and it's much different from the Dark web

Two spiders suspended on a web that is invisible.
W. Drew Senter/Longleaf Photography/Getty Images

What is the Invisible Web? 

Did you know that there is a vast amount of data that search engines won't show you without a specific search? The term "invisible web" mainly refers to the vast repository of information that search engines and directories don't have direct access to, like databases.

Unlike pages on the visible Web (that is, the Web that you can access from search engines and directories), information in databases is generally inaccessible to the software spiders and crawlers that create search engine indexes. Users are able to access most of this information, but only through specific searches that unlock where this information lives. 

How Big is the Invisible Web?

The Invisible Web is estimated to be literally thousands of times larger than the Web content found with general search engine queries. According to Bright Planet, a search organization that specialized in Invisible Web content extraction, the Invisible Web contains nearly 550 billion individual documents compared to the one billion of the surface Web.

The major search engines - Google, Yahoo, Bing - don't bring back all the "hidden" content in a typical search, simply because they can't see that content without specialized search parameters and/or search expertise. However, once a searcher knows how to access this data, there's a huge variety of information available. 

Why Is It Called "The Invisible Web"?

Spiders, which are basically small software programs, meander throughout the Web, indexing the addresses of pages they discover. When these software programs run into a page from the Invisible Web, they don't know quite what to do with it. These spiders can record the address, but can't access anything about the information the page contains.

Why? There's a lot of factors, but mainly they boil down to technical barriers and/or deliberate decisions on the part of the site owner(s) to exclude their pages from search engine spiders. For instance, university library sites that require passwords to access their information will not be included in search engine results, as well as script-based pages that are not easily read by search engine spiders.

Why Is The Invisible Web Important?

Many users believe it could be easier to just stick with what can be found with Google or Yahoo. However, it's not always easy to find what you're looking for with a search engine, especially if you're looking for something a bit complicated or obscure.

Think about the Web as a vast library. Most people wouldn't expect to just walk in the front door and immediately find information on the history of paper clips lying on the front desk; they would expect to dig for it. This is where search engines will not necessarily help you but the Invisible Web will.

The fact that search engines only search a very small portion of the web makes the Invisible Web a very tempting resource. There's a lot more information out there than we could ever imagine.

How Do I Use The Invisible Web?

There are many other people that have asked themselves the exact same question and have put together great sites that serve as a launching point into the Invisible Web. Here are some gateways for different subjects:


  • Voice of the Shuttle: Started in 1994, VoS is one of the oldest and largest humanities databases on the Web, offering an exclusive index of content for subject ranging from Anthropology to Technology of Writing.

Specific to U.S. Government

  • University of Michigan Government Documents Center: You'll find reports, statistics, and other documents from all levels of the U.S. government here. Databases offered include Arts, Health Sciences, Social Sciences, and International Studies.
  • A red carpet portal into the depths of the United States government's many entities. Includes government jobs, an A to Z list of governmental agencies, and information on finding grants, loans, and financial assistance.

Health and Science

  • PsycNET: Use this database from the American Psychological Association to find abstracts and entire journals on various psychological topics (fees apply to complete journal entries, abstracts are usually free).
  • Healthfinder: Professionally vetted information from over one thousand different health databases on the Web. A wide range of resources on a dizzying array of health topics is available here.
  • RXList: If you're looking for reliable drug information, then this database for prescription drugs, medications, and pill identifications is for you. Offers an A to Z index, image collections, and a dictionary of terms.


  • The Virtual Library is simple and easy to use, with annotated subject links. I especially appreciate the annotations because it helps rule out extraneous search time.

What About Other Invisible Web Resources?

There are many, many sites that are set up to dig into the Invisible Web. Most of the information on the Invisible Web is maintained by academic institutions and has a higher quality than search engine results. There are "academic gateways" that can help you find this information. To find nearly any educational resource on the Web, simply type in this search string to your favorite search engine: "subject I'm looking for"

Your search will return with only .edu-related sites. If you have a particular school in mind that you'd like to search, use that school's URL in your search: "subject I'm looking for"

Frame your subject within quotations if it is more than two words; this lets the search engine that you're using know that you want to find those two words right next to each other. Learn more about search tricks to become more proficient in your web searches.

The Bottom Line About The Invisible Web

The Invisible Web offers a vast array of resources on anything you can possibly think of. The links highlighted in this article barely begin to touch the vast resources available on the Invisible Web. As time goes on, the Invisible Web will only get bigger, and that's why it's a good idea to learn how to explore it now.