Web
Scraping &
crawling
w/ R scraper
(Direct) web scraping using R/python
• Why scrap directly and automatically?
• API is usually limited re: what you can get from it
• Services/API calls can be change (often) without notice
• In contrast, direct scraping could be an option as long as you are
interested in some information displayed in a browser
• but you need to repeat a certain task many times.
(Direct) web scraping using R/python
• Web scraping is annoying and error-prone, so it should not
be your first option but rather your last resort.
• You need to know the structure of a web page, and how to
make a scrapper in R or in Python
When there is no API…
• Directly scraping what you can see in the web browser can
be an option
• Instead of copy-pasting, write a scraper that does the job
for you:
Empirical approach
• Scrap / download articles
• Preprocess full texts à submit to text analysis workflow
Cf. Python scraper
• R scraper is very similar
• (We will focus on R today)
Source code - overview
• Webpages include more than what is immediately visible
• Text, images, links, buttons, apps, etc.
• But also hosts codes for structure, style, and functionality –
interpreted by browser, and then displayed for humans
• HTML for structure
• CSS for style
• Javascript for functionality, if any
Source code - overview
• Webpages include more than what is immediately visible
• Web scraping requires working with the source code
• Even when scraping what is already visible
• To choose one or more desired parts of visible info
• Source code also offer more – invisible – data to be scraped
• E.g., URLs hidden under text or links
Source code –
plain text
• https://www.scrapethissite.com/pages/simple/
• The ctrl + U (windows) / option + Cmd + U (Mac)
displays the source code
Source code DOM
• Any modern web browser offer putting the code
in structured view, known as DOM (document
object Model)
• F12 (or Tools – Developer – Developer tools)
Web Definitions: HTML Document vs DOM
HTML Document
DOM (Document Object Model)
An HTML document is a static document
that describes the structure of a web page.
The DOM is a dynamic HTML object that
describes the structure of a web page in
its current state (often interactively).
What a web crawler sees (usually)
What a web browser sees
Web Definitions: Element, Tag, & Attribute
Element:
Tags:
Attributes:
Element: substantive information along with structure, such as headers, paragraphs, lists, tables, images, etc.
Most elements have opening and closing tags, such as <a>…</a>, or <p>…</p> wrapped around its content
Tags can have attributes that define id or styles, sometimes functionalities (such as hyperlink)
Web Definitions: HTML Document vs DOM
HTML Document (static)
DOM (Document Object Model) (dynamic)
An HTML document is a static document
that describes the structure of a web page.
The DOM is a dynamic HTML object that
describes the structure of a web page in its
current state (often interactively).
Static pages are those that display the same source code to all visitors
• Every visitor sees the same content at a given url
Scraping static pages
• https://www.scrapethissite.com/pages/simple/ is a static page
• Static pages are scraped typically in two steps
• The rvest package can handle both steps
• Get the source code into R directly (only interaction takes place)
• Extract the exact information from the source code
• With the rvest, using “selectors” for that information
Static pages – rvest – get source code
• Use the read_html function to get source code into R
• We still need to select the exact information we need by specifying HTML element
Selecting exact elements
• From the downloaded source code, we can get HTML elements
• Two versions:
• Singular one gets the first instance, plural one gets all instances of same kind
• Need to specify a selector, CSS, or xpath
Finding selectors
Activate element inspector, click the
information on page, and it will highlight the
corresponding code on the html
We see that the country name is listed with
h3 tag, so let’s try it
html_elements
html_text
• Get the contents (text) of one or more html elements
• For the elements already chosen
• With html_elements function
• Note that:
• There are two versions of the same function
• html_text returns verbatim (include some code fragment/breaks/space)
• html_text2 returns plain text as we see
html_text
Another example
Another example
• Right click – Copy – Copy
selector
• This returns:
• #countries > div > div:nthchild(4) > div:nth-child(1)
> div > span.country-capital
html_text
Little exercise
• How to make a data frame containing:
• Country name
• Capital
• Population
• Area
Another example
• https://www.scrapethissite.com/pages/forms/
html_table
• Use the html_table function to get text content of table elements
html_attr
• Get one or more attributes of elements selected
• Common attrs of an element is href (internal link)
• https://www.scrapethissite.com/pages/forms/ contain hyperlinks for internal
pagenation
Element selector
This returns:
#hockey > div > div.row.pagination-area >
div.col-md-10.text-center > ul > li:nth-child(1)
> a
Element selector
• ul stands for unordered list
• Has class “pagenation”
• Has attribute a (anchor) with href (hyperlink reference) to ?page_num=1
• This translates in rvest to:
All links for pagenation
url_absolute function converts relative paths (relative to the base address) to absolute paths
Crawling
• Often data is spread across different pages, where you need to perform iterative
operations retrieving similar information many times
• Create a scraper in a way that:
• Visit pages one by one (by identifying all links to be visited)
• Collect and save the information needed
• Write these variables (or tables) into a list
• Convert list to a single data frame
Working example
• This pagenation link contains each table
• Then we can combine this with html_table for each pagenation to get each table
Rudimentary crawler
Scraping
dynamic pages
Dynamic page scraping - Overview
• Dynamic pages are ones that display custom content
• different visitors might see different content on the same page
• Often depending on clicks, scrolls, filling in forms, etc (=human inputs)
• Scraping dynamic pages requires three steps:
• Create: the desired instances of the dynamic page as if humans browse
• Get
• Extract
Dynamic pages with RSelenium
• A package that integrates Selenium WebDriver into R
• Selenium enable us to control web browser programically via R script
• Opening a browser and navigate to page
• With elements on a webpage, it could open, click, or interact with it
• Syntax for Rselenium is bit unusual compared to other packages, beware of
differences!
RSelenium: Requirement
• R/Rstudio
• Install Java SDK on your computer
• Download Java SDK from https://www.oracle.com/java/technologies/downloads/
• Requires restarting any browser that you might have open
• Install Chrome Testing and download chromedriver
• Install Rselenium via install.package(“Rselenium”)
https://www.oracle.com/java/technologies/downloads/; Most up-to-date version as of today is JDK 22
https://www.oracle.com/java/technologies/downloads/; Most up-to-date version as of today is ver. 21
On OSX machine
• Bring up “Terminal.app” (via
spotlight search, or go to
Applications/Utilities/Terminal)
• Type: sudo R CMD javareconf
• If you see the following message,
then you’re good to go
on Windows machine
• Bring up CMD (command prompt) with administrator
privilege
on Windows machine
• Goes to the default installation location of JDK (C:\Program
Files\Java\jdk-22) and copy the path from the explorer
on Windows machine
• Bring up “Run” window (Win + R), type “sysdm.cpl”
on Windows machine
• This brings up “system
properties”. Go to “Advanced”
tap, and click “Environment
variable”
on Windows machine
• Make new system variable, set
name as “JAVA_HOME” and its
value as the path name of the
jdk
on Windows machine
• Restart Rstudio, and type:
• Sys.getenv(“JAVA_HOME”)
Chrome Testing app & Chromedriver
Go to: https://googlechromelabs.github.io/chrome-for-testing/
Look for stable version of Chrome testing app for your OS, download BOTH chrome testing app and driver!
Selenium Stand-alone server
• https://seleniumrelease.storage.googleapis.com
/index.html?path=4.0/
• Download 4.0.0-alpha-1.jar
After downloading the files
• Unzip your files, place all files to a single location
Copy and paste this
to your aplication
folder!
After downloading the files
• Before we actually starts, for OSX users:
(1) Open up Terminal.app
(2) Navigate to the folder
you placed the files
(3) Type this command
into the Terminal
Initiating a server
• Open up terminal (or CMD) and navigate to the folder where all the files are located:
• Type into Terminal (or CMD):
• OSX:
• java -Dwebdriver.chrome.driver="chromedriver" -jar selenium-server-standalone-4.0.0-alpha-1.jar -port
4450
• Windows:
• java -Dwebdriver.chrome.driver="chromedriver.exe" -jar selenium-server-standalone-4.0.0-alpha-1.jar port 4450
• If you see the message “Selenium Server is up and running” then it is good to go
Connecting to a server
•
Go to Rstudio and execute the following command:
•
You will see the new chrome window will appear with the message says that it is controlled by automatic testing
software:
•
Now you can control chrome via R by calling remDr$navigate, remDr$goBack, etc.
Starting a server
• Separate the client and server as different objects
• Note that:
• rsDriver() creates a client (browser) and server (backend program interact with R)
WSP scraper