(ClerkOfCourse)
Katniss-Melb
Jason Kregting (5nacks)
Tom (humanDecoded)
lowprivs
Document Version 1.1
OVA version 2020.2
Trace Labs is a Not-For-Profit organization with a mission to crowdsource the collection
of Open Source Intelligence (OSINT) to generate new leads on missing persons cases.
The missing persons issue is worsening which requires modern and scalable solutions
at various levels to help mitigate risk to society.
We leverage our own custom CTF platform that enables the collection of OSINT to
power crowdsourced Capture the Flag (CTF) events known as the “OSINT Search Party
CTF”. OSINT refers to the collection, processing, and analysis of publicly available data
such as social media, forums, government records, and even the dark web.
Trace Labs has taken the traditional CTF competition that we see in the information
security community where participants hack into intentionally vulnerable servers to
obtain “Flags” for points and evolved it into a real-life exercise where the participants’
contributions have real-world impact and the potential to enhance public safety.
Since its inception in 2018, Trace Labs has:
•
•
•
•
•
•
Organized 30 CTFs globally
Worked on 250+ missing persons cases
Collected 30,000+ OSINT submissions from our crowdsourced community
Brought together 2500+ contestants in our CTFs
Brought together 500+ volunteer CTF Judges
Worked with 10+ Law Enforcement Agencies
Contents
......................................................................................................................................................... 3
Trace Labs OSINT Virtual Machine (VM) .................................................................................... 5
// Introduction ............................................................................................................................ 5
// Licenses .................................................................................................................................. 5
// System Requirements ........................................................................................................... 5
// Distribution Tools and Features........................................................................................... 6
How to install................................................................................................................................. 8
// Download the OVA ................................................................................................................. 8
// Import the OVA file into the virtualization software ........................................................... 9
// Start the Trace Labs OSINT VM .......................................................................................... 11
How To / Troubleshooting ......................................................................................................... 12
// The virtual machine is running slowly! ............................................................................. 12
// I can’t install VMWare or VirtualBox on Windows 10 ...................................................... 12
// Intel/AMD Virtualization not enabled in BIOS ................................................................... 12
// The screen is hard to read .................................................................................................. 13
// Failed to import appliance. Code: NS_ERROR_INVALID_ARG(0x80070057) .............. 13
Tools Overview ........................................................................................................................... 14
// Browsers ............................................................................................................................... 14
// Data Analysis Tools ............................................................................................................. 15
// Domains ................................................................................................................................ 16
// Downloaders ......................................................................................................................... 17
// Email ...................................................................................................................................... 18
// Frameworks .......................................................................................................................... 19
// Phone Numbers .................................................................................................................... 20
// Social Media.......................................................................................................................... 21
// Usernames ............................................................................................................................ 22
Trace Labs OSINT Virtual Machine (VM)
// Introduction
The Trace Labs team has set out to create a specialized OSINT VM specifically to
bring together the most effective OSINT tools and customized scripts we saw being
used during our Search Party CTF’s. Inspired by the popular Buscador VM by
Michael Bazzell, the Trace Labs OSINT VM was built in a similar way, to enable
OSINT investigators participating in the Trace Labs Search Party CTFs a quick way
to get started and have access to the most popular OSINT tools and scripts all neatly
packaged under one roof.
We are continuing to build upon the Trace Labs OSINT VM and welcome any and all
feedback. Our goal with this project is to create an OSINT focused VM that provides
security, stealth, and the ability to easily save digital forensic evidence during an
investigation all within an easy to use package.
// Licenses
This Linux Distribution is a modified version of Kali Linux which is developed by
Offensive Security and contains free and non-free packages. See
https://www.kali.org/docs/policy/kali-linux-open-source-policy/ for licensing details.
Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
// System Requirements
The virtual machine is currently pre-allocated with 4G of RAM, 4 CPU cores and 40G
disk space. It requires a 64-bit processor.
Your computer should have the following specifications:
• OS: Windows 10 x64 / Mac OS X / Linux Distribution x64
• Processor: Intel Core i3 2.5 Ghz or AMD Phenom II 2.6 Ghz or greater
• Memory: 8 Gigabytes of RAM
• More than 40G of disk space free
If there are not enough resources allocated to the Virtual Machine it will run slowly or
hang, particularly when running multiple browser tabs.
// Distribution Tools and Features
The distribution includes the following tools and features:
Domains
• Sublist3r
Downloaders
• Browse Mirrored
Websites
• Metagoofil
• Spiderpig
• WebHTTrack
Website Copier
• Youtube-DL
Browsers
Email
•
•
•
Chromium Web Browser
Firefox ESR
Tor Browser
Data Analysis
•
•
•
•
•
DumpsterDiver
Exifprobe
Exifscan
Photon
Stegosuite
Phone Numbers
•
•
OSINT-Search
PhoneInfoga
•
•
•
•
•
Buster
H8mail
Infoga
OSINT-Search
theHarvester
Frameworks
•
•
•
•
•
•
FinalRecon
Little Brother
recon-ng
sn0int
Spiderfoot
WikiLeaker
Social Media
•
•
Instaloader
Twint
Usernames
Configuration Settings
Firefox
• Delete cookies/history on shutdown
• Privacy protection (block
mic/camera/geo)
• OSINT Bookmarks
•
•
Sherlock
WhatsMyName
// Support
This customised Kali Linux distribution is supported by the community and does not
come with any official support. Please visit the following communities to get support.
Trace Labs Community
Trace Labs has a Slack page (www.tracelabs.org) and a channel #questions where
you can ask about OSINT methods and tools.
Offensive Security
Offensive Security provides a forum for support with the Kali Distribution.
https://www.kali.org/community/
How to install
// Install virtualization software
To use the Trace Labs OSINT Operating System (OS), you will need to use a Virtual
Machine (VM). It is suggested that you install the OS in a VM instead of installing it as
your computer’s operating system. You can easily create a snapshot before you start
your investigations and rollback to it once the CTF event is over.
You can use VirtualBox or VMWare.
If you don’t have a virtualization software, you can download the latest from
VirtualBox here:
https://www.virtualbox.org/wiki/Downloads
If you have VMWare installed, the instructions on how to import the OVA file is found
in the sections below.
// Download the OVA
Obtain the OVA from this location https://www.tracelabs.org/trace-labs-osint-vm/
Once downloaded check the hash of the file to ensure that the file you downloaded
hasn’t been tampered with.
If you have a program that can check file hashes, such as 7-zip this can be done
within in Windows Explorer as per the screenshot below:
// Import the OVA file into the virtualization software
Virtual Box
You can find instructions on how to do this here:
https://docs.oracle.com/cd/E26217_01/E26796/html/qs-import-vm.html
VMWare Fusion
Step 1: Go to File>Import. Choose the OVA file you’ve downloaded.
Step 2: Once you’ve selected the OVA file, click Continue.
Step 3: Save the virtual machine.
Note: If you encounter the message that the import failed because the OVA file did not
pass OVF specification conformance or virtual hardware compliance checks, just click
Retry.
Step 4: If you want to change the default virtual machine settings, click Customize
Settings. Otherwise, just click Finish.
VMWare Workstation Pro
Step 1: Go to File> Open. Select the OVA you have downloaded.
Step 2: Choose the OVA file you’ve downloaded.
Note: If you encounter the message that the import failed because the OVA file did
not pass OVF specification conformance or virtual hardware compliance checks, just
click Retry.
Step 3: Wait for a few minutes for the importing to complete. Once it is
completed, you will see it saved in your VMWare Workstation and you can use the
green play button to start it.
// Start the Trace Labs OSINT VM
Virtual Box
• Click on the Start button on VBox to begin.
VMWare Fusion
•
Click on the play button to start your newly imported VM. The other option is to
click on File>Open and Run and select the VM you have just imported.
VMWare Workstation Pro
•
Click on the play button to start your newly imported VM.
Login to the Virtual Machine
•
Use the following credentials and then hit enter:
Username: osint
Password: osint
How To / Troubleshooting
// The virtual machine is running slowly!
See the following links to increase the amount of resources in the virtual machine so
that you can run more applications concurrently.
VMWare: https://kb.vmware.com/s/article/1004059
VirtualBox: https://docs.bitnami.com/virtual-machine/faq/administration/increasememory/
// I can’t install VMWare or VirtualBox on Windows 10
Windows 10 has a new feature called Credential Guard which stops VMware being
installed. You may want to refer to the following Microsoft article.
https://support.microsoft.com/en-au/help/3204980/virtualization-applications-do-notwork-together-with-hyper-v-device-g
// Intel/AMD Virtualization not enabled in BIOS
You may get an error such as this when trying to power on a virtual machine. If so, it
means you need to enable virtualization in your BIOS.
https://www.howtogeek.com/213795/how-to-enable-intel-vt-x-in-your-computers-biosor-uefi-firmware/
// The screen is hard to read
On high definition monitors the virtual machine may appear to be hard to read.
Please refer to this guide:
https://www.kali.org/docs/general-use/hidpi/
// Failed to import appliance. Code: NS_ERROR_INVALID_ARG(0x80070057)
If you encounter a message in VirtualBox that it failed to import appliance, there is
not enough space on the disk. Refer to the VM specs and make sure there’s enough
hard disk space.
Tools Overview
// Browsers
The Chromium and Firefox browsers are installed. When you first open these browsers,
you’ll see the following browser extension info page loaded:
1. Privacy Badger – this extension automatically blocks invisible trackers.
2. Add0n Media Tools – this extension detects media resources from web
pages. This can be used to grab media files like video or photos from a webpage.
Aside from the above, the EFF HTTPS Everywhere extension is also installed to make
sure that communications from the browser with major websites are encrypted.
When you click on the Tor Browser for the first time from the applications menu, this will
initiate the download and installation of the Tor browser.
In the Firefox browser, you will find the OSINT Bookmarks in the toolbar. It includes several
websites that the TraceLabs volunteers have used in their OSINT investigations.
// Data Analysis Tools
The following tools are installed in the VM:
1. DumpsterDiver – this command line (CLI) tool will analyze a big volume of
data for hardcoded secrets likes keys.
2. Exifgrep – this is a shell script that reports on the EXIF data found in an
image.
3. Exifprobe – this tool will read image files and reports on the structure of
the files and the metadata contained within the files.
4. Photon – this is an CLI-based OSINT web crawler.
5. Stegosuite – this is a steganography tool that can be used to hide
information in image files.
// Domains
Sublist3r – is a CLI-based tool that will enumerate subdomains of websites using
OSINT.
// Downloaders
The following tools are installed in the VM:
1. WebHTTrack Website Copier – this GUI-based tool will back up complete
websites for offline access. Once the offline copies have been made, you can browse
the mirrored websites.
2. Metagoofil – this CLI tool will extract metadata of public documents
available in the target website.
3. Spiderpig – this CLI tool will harvest metadata by spidering or crawling a
website first, then downloading the documents before parsing out data.
4. Youtube-DL – this CLI tool will download videos from YouTube.com and
other sites.
// Email
The following tools are installed in the VM:
1. Buster – this tool is for finding information based on email or username. It
will get social accounts of an email, breaches involving an email, domains registered
using an email, and generate potential email and usernames of a person.
2. H8mail – this tool is for email information and password lookup using
different data breach and reconnaissance services.
3. Infoga – this tool is for gathering email account information from different
public sources.
4. OSINT-Search – this tool will search public data repositories using email
addresses, phone numbers, domains, IP addresses or URLs.
5. theHarvester – this tool will gather email, names, subdomains, IPs and
URLs using multiple public data sources.
// Frameworks
The following tools are installed in the VM:
1. FinalRecon – this tool is for doing web reconnaissance. It provides header
information; SSL certificate information; results of whois lookups, DNS enumeration,
sub-domain enumeration, traceroute and others.
2. LittleBrother – this tool is an information collection tool for doing research
on a French, Swiss, Luxembourger or Belgian person.
3. recon-ng – this is a reconnaissance framework that can be used to
conduct open source web-based reconnaissance.
4. sn0int – this is a semi-automatic OSINT framework that will gather
intelligence on a given target.
5. Spiderfoot – this is an OSINT automation tool that gathers intel about IP
addresses, domains, e-mail addresses and research the targets from many data
sources.
6. WikiLeaker – this is a scraper for domains.
// Phone Numbers
The following tools are installed in the VM:
1. OSINT-Search – this tool will search public data repositories using email
addresses, phone numbers, domains, IP addresses or URLs.
2. PhoneInfoga – this tool will check if the phone number exists and gather
standard information such as country, line type and carrier. It will also check for
reputation reports.
// Social Media
The following tools are installed in the VM:
1. Instaloader – this is a tool that will download various types of data from an
Instagram profile.
2. Twint – this is a tool that will scrape tweets from Twitter profiles without
using the Twitter API.
// Usernames
The following tools are installed in the VM:
1. Sherlock – this tool will find usernames across different social networks.
2. WhatsMyName – this is a standalone script that will look up a single
username.
2.
https://www.kali.org/docs/general-use/hidpi/