Web scraping with Beautiful Soup

Author

Marie-Hélène Burle

The internet is a trove of information. A lot of it is publicly available and thus suitable for use in research. Extracting that information and putting it in an organized format for analysis can however be extremely tedious.

Web scraping tools allow to automate parts of that process and Python is a popular language for the task.

In this workshop, I will guide you through a simple example using the package Beautiful Soup.

HTML and CSS

HyperText Markup Language (HTML) is the standard markup language for websites: it encodes the information related to the formatting and structure of webpages. Additionally, some of the customization can be stored in Cascading Style Sheets (CSS) files.

HTML uses tags of the form:

<some_tag>Your content</some_tag>

Some tags have attributes:

<some_tag attribute_name="attribute value">Your content</some_tag>

Examples:

Site structure:

<h2>This is a heading of level 2</h2>
<p>This is a paragraph</p>

Formatting:

<b>This is bold</b>
<a href="https://some.url">This is the text for a link</a>

Web scrapping

Web scraping is a general term for a set of tools which allow for the extraction of data from the web automatically.

While most of the data on the internet is publicly available, it is illegal to scrape some sites and you should always look into the policy of a site before attempting to scrape it. Some sites will also block you if you submit too many requests in a short amount of time, so remember to scrape responsibly.

Example for this workshop

We will use a website from the University of Pennsylvania that gives access to three million free online books.

Our goal is get a list of books by Proust available in that database. We can go to the online books author search page and enter “Proust” in the search box, it gets us to the URL https://onlinebooks.library.upenn.edu/webbin/book/search?author=proust&amode=words which contains the data we are interested in. From here on, you could copy-paste the data one line at a time, but it is rather tedious. Instead, we will automate this task.

Install packages

For this section, you need to install the following packages in a virtual environment (you can create a new one or activate the one you used in the previous days):

requests
beautifulsoup4

requests allows you to send commands to a web server. In our case, we use it to get the HTML of the page we are interested in while beautifulsoup4 makes it easy to pull data out of HTML files.

Load packages

Let’s load the packages that will make scraping websites with Python easier:

import requests                 # To download the HTML data from a site
from bs4 import BeautifulSoup   # To parse the HTML data

Let’s create a string with the URL we want to scrape:

url = "https://onlinebooks.library.upenn.edu/webbin/book/search?author=proust&amode=words"

First, we send a request to that URL and save the response in a variable called r:

r = requests.get(url)

Let’s see what our response looks like:

<Response [200]>

If you look in the list of HTTP status codes, you can see that a response with a code of 200 means that the request was successful.

Explore the raw data

To get the actual content of the response as unicode (text), we can use the text property of the response. This will give us the raw HTML markup from the webpage.

Let’s print the first 400 characters:

print(r.text[:400])

<!DOCTYPE html>
<html lang="en"><head>
<meta charset="utf-8">
<link rel="stylesheet" type="text/css" href="https://onlinebooks.library.upenn.edu/olbp.css" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Search Results | The Online Books Page</title></head>
<body>
<header>
<h1><a href="https://onlinebooks.library.upenn.edu/" class="logolink">The Online Books Page</a

Parse the data

The package Beautiful Soup transforms (parses) such HTML data into a parse tree, which will make extracting information easier.

Let’s create an object called mainpage with the parse tree:

data = BeautifulSoup(r.text, "html.parser")

html.parser is the name of the parser that we are using here. It is better to use a specific parser to get consistent results across environments.

We can print the beginning of the parsed result:

print(data.prettify()[:400])

<!DOCTYPE html>
<html lang="en">
 <head>
  <meta charset="utf-8"/>
  <link href="https://onlinebooks.library.upenn.edu/olbp.css" rel="stylesheet" type="text/css"/>
  <meta content="width=device-width, initial-scale=1.0" name="viewport"/>
  <title>
   Search Results | The Online Books Page
  </title>
 </head>
 <body>
  <header>
   <h1>
    <a class="logolink" href="https://onlinebooks.library.upenn

The prettify method turns the BeautifulSoup object we created into a string (which is needed for slicing).

It doesn’t look any more clear to us, but it is now in a format the Beautiful Soup package can work with.

For instance, we can get the HTML segment containing the title of the page with three methods:

using the title tag name:

data.title

<title>Search Results | The Online Books Page</title>

using find to look for HTML markers (tags, attributes, etc.):

data.find("title")

<title>Search Results | The Online Books Page</title>

using select which accepts CSS selectors:

data.select("title")

[<title>Search Results | The Online Books Page</title>]

Now, this method returns a bs4 ResultSet object which behaves like a list (you can see that the output is in square brackets). To get the content, we need to index it out of the object:

mainpage.select("title")[0]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[11], line 1
----> 1 mainpage.select("title")[0]

NameError: name 'mainpage' is not defined

find will only return the first element. find_all will return all elements. select will also return all elements. Which one you chose depends on what you need to extract. There often several ways to get you there.

Below are other examples of data extraction.

The beginning of the parsed data:

data.head

<head>
<meta charset="utf-8"/>
<link href="https://onlinebooks.library.upenn.edu/olbp.css" rel="stylesheet" type="text/css"/>
<meta content="width=device-width, initial-scale=1.0" name="viewport"/>
<title>Search Results | The Online Books Page</title></head>

The first link in the data:

data.a

<a class="logolink" href="https://onlinebooks.library.upenn.edu/">The Online Books Page</a>

The first 5 links (returned as a list):

data.find_all("a")[:5]

[<a class="logolink" href="https://onlinebooks.library.upenn.edu/">The Online Books Page</a>,
 <a href="https://onlinebooks.library.upenn.edu/webbin/book/browse?type=lcsubc&amp;key=Proust%2c%20Marcel%2c%201871%2d1922">Online books about this author</a>,
 <a href="https://en.wikipedia.org/wiki/Marcel_Proust">Wikipedia article</a>,
 <a href="https://onlinebooks.library.upenn.edu/webbin/book/lookupid?key=olbp26606"><img alt="[Info]" class="info" src="https://onlinebooks.library.upenn.edu/info.gif"/></a>,
 <a href="https://gutenberg.net.au/ebooks03/0300501.txt"><cite>The Captive</cite> (English translation published 1929)</a>]

or:

data.select("a")[:5]

[<a class="logolink" href="https://onlinebooks.library.upenn.edu/">The Online Books Page</a>,
 <a href="https://onlinebooks.library.upenn.edu/webbin/book/browse?type=lcsubc&amp;key=Proust%2c%20Marcel%2c%201871%2d1922">Online books about this author</a>,
 <a href="https://en.wikipedia.org/wiki/Marcel_Proust">Wikipedia article</a>,
 <a href="https://onlinebooks.library.upenn.edu/webbin/book/lookupid?key=olbp26606"><img alt="[Info]" class="info" src="https://onlinebooks.library.upenn.edu/info.gif"/></a>,
 <a href="https://gutenberg.net.au/ebooks03/0300501.txt"><cite>The Captive</cite> (English translation published 1929)</a>]

Identify relevant markers

The HTML code for this webpage contains the data we are interested in, but it is mixed in with a lot of HTML formatting and data we don’t care about. We need to extract the data relevant to us and turn it into a workable format.

The first step is to find the HTML markers that contain our data. One option is to use a web inspector or—even easier—the SelectorGadget, a JavaScript bookmarklet built by Andrew Cantino.

To use this tool, go to the SelectorGadget website and drag the link of the bookmarklet to your bookmarks bar.

Now, go to the search result page and click on the bookmarklet in your bookmarks bar. You will see a floating box at the bottom of your screen. As you move your mouse across the screen, an orange rectangle appears around each element over which you pass and it give you the HTML tag of that element.

Click on one of the titles by Proust: now, cite appears in the box at the bottom as well as the number of elements selected. The selected elements are highlighted in yellow.

Now you know that there are 125 titles (the number on the JavaScript box).

The cite HTML tag is a semantic element used to define titles of creative work (books, movies, songs, research papers, paintings, plays, websites). It tells search engines and assistive technologies that the wrapped text is a formal reference or attribution to a work.

By default, browsers display the content inside a cite tag in italics.

Look at the page with our search results and notice that the book titles are indeed in italic.

Extract the first title

Now we can use select with the CSS selector cite to extract all the titles.

Let’s create a variable that we call titles:

titles = data.select("cite")

We can confirm that we do indeed have 125 titles:

len(titles)

To get the first one, we index it:

titles[0]

<cite>The Captive</cite>

Your turn:

How would you get the second title?
How about the first five titles?
What is the type of the titles object?

Now, our titles are in those cite tags. To remove them, we can use the get_text method:

titles[0].get_text()

'The Captive'

type(titles[0].get_text())

str

As you can see, we get a string (which is what we want).

Now, we need to apply the get_text method on all the elements of the list titles. For this, we use a for loop.

First, we create an empty list that we can fill with the cleaned titles as the loop runs:

cleaned_titles = []

print(f"cleaned_titles is a {type(cleaned_titles)} of length {len(cleaned_titles)}")

cleaned_titles is a <class 'list'> of length 0

Now we run the loop and fill in our empty list with append:

for title in titles:
    cleaned_titles.append(title.get_text())

Your turn:

Print the last 10 cleaned titles.

cleaned_titles[-10:]

['Tendres stocks',
 'Un amour de Swann.',
 'Un amour de Swann',
 'Un amour de Swann ; avec douze aquarelles par Hermine David.',
 'Within a budding grove',
 'Within a budding grove',
 'Within a budding grove',
 'Within a budding grove',
 'Within a budding grove',
 'Within a Budding Grove']

As you can see, there are a lot of duplicates in our list. To make matters worse, some titles end with a period, others don’t and the capitalization is inconsistent. We need to address this before we can clean the duplicates.

periodfree_titles = [item.replace(".", "") for item in cleaned_titles]

Let’s print the last titles again:

periodfree_titles[-10:]

['Tendres stocks',
 'Un amour de Swann',
 'Un amour de Swann',
 'Un amour de Swann ; avec douze aquarelles par Hermine David',
 'Within a budding grove',
 'Within a budding grove',
 'Within a budding grove',
 'Within a budding grove',
 'Within a budding grove',
 'Within a Budding Grove']

Now we apply the same method to turn everything into lower case:

lowercase_titles = [item.lower() for item in periodfree_titles]

Sanity check:

lowercase_titles[-10:]

['tendres stocks',
 'un amour de swann',
 'un amour de swann',
 'un amour de swann ; avec douze aquarelles par hermine david',
 'within a budding grove',
 'within a budding grove',
 'within a budding grove',
 'within a budding grove',
 'within a budding grove',
 'within a budding grove']

Finally, we can remove the duplicates:

unique_list = list(set(lowercase_titles))

If you sort and print our final list, you can see that it is still far from perfect.

unique_list.sort()
unique_list

['47 unpublished letters from marcel proust to walter berry',
 "a l'ombre des jeunes filles en fleurs",
 "a l'ombre des jeunes filles en fleurs, t",
 'a la recherche du temps perdu',
 'a la sombra de las muchachas en flor',
 'albertine disparue',
 'albertine disparue vol 1 (of 2): à la recherche du temps perdu, tome 7',
 'albertine disparue vol 2 (of 2): à la recherche du temps perdu, tome 7',
 'au bal avec marcel proust,',
 'auf den spuren der verlorenen zeit dritter roman, die herzogin von guermantes',
 'autour de soixante lettres de marcel proust',
 'chroniques',
 'cities of the plain',
 'comment débuta marcel proust',
 'comment parut "du côté de chez swann"',
 'comment parut "du côté de chez swann" lettres de marcel proust',
 'correspondance générale de marcel proust',
 'der weg zu swann',
 'du côté de chez swann',
 'du côté de chez swann',
 'im schatten der jungen mädchen',
 "la bible d'amiens",
 'la prisonnière (sodome et gomorrhe, iii)',
 'la prisonnière--sodome et gomorrhe, iii',
 'la prisonnière (sodome et gomorrhe iii)',
 'la prisonnière (sodome et gomorrhe, iii)',
 'le còté de guermantes, 1 sodome et gomorrhe, 2',
 'le côté de guermantes',
 'le prisonnière (sodome et gomorrhe, iii)',
 'le temps retrouvé',
 'le temps retrouvé',
 'le temps retrouvé tome 1 (de 2) : à la recherche du temps perdu volvii',
 'le temps retrouvé tome 2 (de 2) : à la recherche du temps perdu volvii',
 'les cahiers marcel proust ',
 'les plaisirs et les jours',
 'lettres de marcel proust; comment parut "du côté de chez swann" introd et commentaires par léon pierre-quint',
 'lettres inédites',
 'lettres à madame scheikévitch',
 'marcel proust à dix-sept ans',
 'marcel proust; lettres et conversations',
 'morceaux choisis de marcel proust',
 'oeuvres complètes de marcel proust',
 'oeuvres complètes de marcel proust ',
 'pastiches et mélanges',
 'pastiches et mélanges',
 'por el camino de swann',
 "portraits de peintres : pièces pour piano d'après les poésies de marcel proust",
 'propos de peintre, première série: de david à degas: ingres, david, manet, degas, renoir, cézanne, whistler, fantin-latour, ricard, conder, beardsley, etc préface par marcel proust',
 'quelques lettres de marcel proust à jeanne, simone, gaston de caillavet, robert de flers, bertrand de fénelon',
 'quelques lettres de marcel proust, précédées de remarques sur les derniers mois de sa vie',
 'remembrance of things past',
 'remembrance of things past ',
 'selections from marcel proust',
 'sésame et les lys : des trésors des rois, des jardins des reines',
 'sodome et gomorrhe',
 'sodome et gomorrhe ii',
 'sodome et gomorrhe, 2',
 'sodome et gomorrhe, ii',
 'sodome et gomorrhe, ii ',
 "swann's way",
 'sésame et les lys: des trésors des rois, des jardins des reines',
 'tendres stocks',
 'the captive',
 'the guermantes way',
 'the sweet cheat gone',
 'time regained',
 'un amour de swann',
 'un amour de swann ; avec douze aquarelles par hermine david',
 'within a budding grove',
 'à la recherche du temps perdu ']