What is the invisible web?
 All information that cannot be indexed
using general web search engines
 Also known as the deep internet, deepnet,
or the hidden web
Search Engines
 Programs that search documents for
specified keywords or phrases and returns
a list of documents where the keywords
were found
How Search Engines Work
 Uses a spider program to fetch as many
webpages as possible
 A program called an indexer then reads
these webpages and creates an index,
storing the URL and important content of
webpage
 Each search engine has its own ranking
algorithm that returns results based on
their relevance to the user’s specified
keywords or phrases
Spider Programs
 Starts by searching heavily used servers
and the most popular web pages
 Visits all the links found in the webpages it
reads and sends them to the search engine’s
indexer in a process called web crawling,
which continues automatically
 To be discovered, a webpage must be static
and linked to other pages
 All pages that are indexed by a search
engine’s spider programs are known as the
visible web
The Invisible Web
 Everything not found by a search engine’s
spider program is part of the invisible web
 Most webpages on the internet are not
indexed by search engines
 Some experts estimate that as much as
75% of the entire internet is invisible web
content
Content Found On The
Invisible Web
 Webpages with no links on them, called
disconnected pages
 Password protected webpages
 Webpages generated from databases
 Dynamically generated webpages
 Real-time content
 Webpages that require a registration form
to access
 Webpages with non-html text, or any
coding that a spider program can not
understand
Advantages Of Invisible
Web Content
 Specialized content focus – large amounts
of information focused on an exact subject
 Contains information that might not be
available on the visible web
 Allows a user to find a precise answer to a
specific question
 Allows a user to find webpages from a
specific date or time
So what is actually on the Deep Web,
and how do we get there?
Deep Web (layer two)
- Content that is only available through
specialized anonymizing software (Tor). This
includes anonymity networks and darknets
(anonymous P2P). This are is a sort of safehaven for speech and content which has been
banned from the surface web.
- examples: Anonymous chat/imageboards.
CP, drugs, illeagal transactions, weapons, DRM
removal software, pirated software/media/
books, black markets
Accessing the Deep Web
.onion:
-onion is a domain host suffix designating an anonymous
hidden service reachable via the Tor network.
-The purpose of using such a system is to make both the
information provider and the person accessing the
information more difficult to trace, whether by one
another, by an intermediate network host, or by an
outsider.
-.onion adresses are 16-character non-mneumonic
hashes, compromised of alphabetic and numeric strings.
-The "onion" name refers to onion routing, the technique
used by Tor to achieve a degree of anonymity.
Accessing the Deep Web
Tor:
-” The Tor software protects you by bouncing your
communications around a distributed network of relays run by
volunteers all around the world: it prevents somebody watching your
Internet connection from learning what sites you visit, it prevents the
sites you visit from learning your physical location, and it lets you
access sites which are blocked.” – www.torproject.org
-Tor is a network that supports onion routing; a way to help
make your traffic anonymous. Because the Deep Web is
compromised of information that doesn’t show up on search
engines, or has no domain name registry, you must know exactly
where you are going to get there.
The first step is to download, install, and run the Tor Bundle. This will bring up your new
anonymous browser (through Firefox) in order to get started.
Guided Journey Through the Deep Web
The deep web is vast, as we covered before, and most is private.
Here we will go through some examples of websites on the Deep
Web, how online currency works, and the various resources the Deep
Web encompasses (not all entirely legal)
This content can be disturbing. Obviously with great resources such
as free library databases, and banned books, comes information that
is considered scat. Being a safe-haven from the censorship of the
surface web, there is a lot of CP, arms/drug trade, and other illegal
information on the Deep Web. It’s up to you as an individual to steer
clear of what you don’t want to see.
In addition, one must also be skeptical. There are even sites
dedicated to contract killing, but one can’t be sure that these
websites are even serious.
The next step is to access the Hidden Wiki, which most people consider the home page of the Deep Web.
Here you can begin your journey and discover many different types of sites and networks, ranging from tame
to very illegal. There are black market type sites, hacking information sites, huge file databases, political
advocacy sites, and even sites to hire people to engage in illegal activity.
One of the other “main pages” of the Deep Web is TorDir. TorDir is a site that uses a special crawler for the
Deep Web, so that it may act similar to Google in that it categorizes web sites, and allows you to view many
different and obscure .onion sites that fall into each category.
Through the Hidden Wiki you can find pages like this, which are semi-organized lists of different .onion sites.
Many different sites are listed and separated based on function, such as buying/selling/trading,
communication, hacking, or intel exchange.
Currency of the Deep Web
Bitcoins:
-an electronic cash system using peer-to-peer networking, digital
signatures and cryptographic proof to enable irreversible payments
between parties without relying on trust. Payments are issued and
transferred by the Bitcoin network.
-Participants begin using Bitcoin by first acquiring a Bitcoin wallet and
one or more Bitcoin addresses. Bitcoin addresses are used for receiving
Bitcoins, similar to how e-mail addresses are used for receiving e-mail.
-In April 2012, 1 BTC traded at around $4.80. Taking into account the total
number of Bitcoins in circulation, the market capitalization of the Bitcoin
network stands at over 40 million USD
This is an example of one of the many online black markets. This one did not require a paid
membership. There are many different ways to spend bit coins, such as on apparel, money
transfers, drugs, books, and even digital goods.
This is an example of someone’s own personal page, with content picked out due to it’s
controversial nature. Here you can find anything from banned readings, to trefelling. There is
content on making one’s own explosives, and many different military type documents and
guidebooks.
There is a huge portion of the Deep Web that is not malicious at all. This is a huge database of
banned, black listed, and forbidden books which you can read and download. Books have been
burned in the past and banned for their controversial subject matter, and this is a way for
history to be preserved.
Another scholarly database, this .onion site is a compilation of financial, business, real estate,
and even marketing tools and texts free to anyone who wants to download them. The Deep
Web is a great archive for educational material.
The Deep Web is full of malicious content. On this Russian-based community hacking website,
one can purchase credit card information, PayPal account information, bank accounts, and even
the service of DDoSing a website. This is when the website is overloaded with information sent
to it, and eventually is forced to shut down.
Dangers of the Deep Web
If you decide to look into the Deep Web on your own volition,
make sure to be careful.
-Have anti-virus protection.
-Be intelligent about what links you click. The Deep
Web is a haven for Phishers.
-If you don’t want to see any disturbing images or
content, simply browse as text-only.