Python Spider Tutorial

Python Scrapy tutorial for beginners – 01 – Creating your first …

Learn how to fetch the data of any website with Python and the Scrapy Framework in just minutes. On the first lesson of ‘Python scrapy tutorial for beginners’, we will scrape the data from a book store, extracting all the information and storing in a file.
In this post you will learn:
Prepare your environment and install everythingHow to create a Scrapy project and spiderHow to fetch the data from the HTMLTo manipulate the data and extract the data you wantHow to store the data into a, and file
Video version of this lesson
Preparing your environment and installing everything
Before anything, we need to prepare our environment and install everything.
In Python, we create virtual environments to have a separated environment with different example, Project1 has Python 3. 4 and Scrapy 1. 2, and Project2 Python 3. 7. 3. As we keep separated environments, one for each project, we will never have a conflict by having different versions of packages.
You can use Conda, virtualenv or Pipenv to create a virtual environment. In this course, I will use pipenv. You only need to install it with pip install pipenv and to create a new virtual environment with pipenv shell.
Once you are set, install Scrapy with pip install scrapy. That’s all you need.
Time to create the project and your spider.
Base image provided by Vecteezy
Creating a project and a spider – And what they are
Before anything, we need to create a Scrapy project. In your current folder, enter:
scrapy startproject books
This will create a project named ‘books’. Inside you’ll find a few files. I’ll explain them in a more detailed post but here’s a brief explanation:
books/
<-- Configuration file (DO NOT TOUCH! ) tutorial/ <-- Empty file that marks this as a Python folder <-- Model of the item to scrap <-- Scrapy processing hooks (DO NOT TOUCH) <-- What to do with the scraped item <-- Project settings file spiders/ <-- Directory of our spiders (empty by now) After creating a project, navigate to the project created (cd books) and once inside the folder, create a spider by passing it the name and the root URL without ‘www’: scrapy genspider spider Now we have our spider inside the spider folder! You will have something like this: # -*- coding: utf-8 -*- import scrapy class SpiderSpider(): name = 'spider' allowed_domains = [''] start_urls = [''] def parse(self, response): pass First, we import scrapy. Then, a class is created inheriting ‘Spider’ from Scrapy. That class has 3 variables and a method. The variables are the spider’s name, the allowed_domains and the start_URL. Pretty self-explanatory. The name is what we will use in a second to run the spider, allowed_domains limit the scope of the scraping process (It can’t go outside any URL not specified here) and start_urls are the starting point of the scrapy spider. In this case, just one. The parse method is internally called when we start the Scrapy spider. Right now has only ‘pass’: It does nothing. Let’s solve that. How to fetch data from the HTML We are going to query the HTML and to do so we need Xpath, a query language. Don’t you worry, even if it seems weird at first, it is easy to learn as all you need are a few functions. Parse method But first, let’s see what we have on ‘parse’ method. Parse it’s called automatically when the Scrapy spider starts. As arguments, we have self (the instance of the class) and a response. The response is what the server returns when we request an HTML. In this class, we are requesting and in response we have an object with all the HTML, a status message and more. Replace “pass” with ‘print()‘ and run the spider: scrapy crawl spider This is what we got: Between a lot of information, we see that we have crawled the start_url, got a 200 HTTP message (Success) and then the spider stopped. Besides ‘status’, our spider has a lot of methods. The one we are going to use right now is ‘xpath’. Our first steps with Xpath Open the starting URL, and right-click -> inspect any book. A side menu will open with the HTML structure of the website (if not, make sure you have selected the ‘Elements’ tab). You’ll have something like this:
We can see that each ‘article’ tag contains all the information we want.
The plan is to grab all articles, then, one by one, get all the information from each book.
First, let’s see how we select all articles.
If we click on the HTML the side menu and press Control + F, the search menu opens:
At the bottom-right, you can read “Find by string, selector or Xpath”. Scrapy uses Xpath, so let’s use it.
To start a query with Xpath, write ‘//’ then what you want to find. We want to grab all the articles, so type ‘//article’. We want to be more accurate, so let’s grab all the articles with the attribute ‘class = product_pod’. To specify an attribute, type it between brackets, like this: ‘//article[@class=”product_pod”]’.
You can see now that we have selected 20 elements: The 20 initial books.
Seems like we got it! Let’s copy that Xpath instruction and use it to select the articles in our spider. Then, we store all the books.
all_books = (‘//article[@class=”product_pod”]’)
Once we have all the books, we want to look inside each book for the information we want. Let’s start with the title. Go to your URL and search where the full title is located. Right-click any title and then select ‘Inspect’.
Inside the h3 tag, there is an ‘a’ tag with the book title as ‘title’ attribute. Let’s loop over the books and extract it.
for book in all_books:
title = (‘. //h3/a/@title’). extract_first()
We get all the books, and for each one of them, we search for the ‘h3’ tag, then the ‘a’ tag, and we select the @title attribute. We want that text, so we use ‘extract_first‘ (we can also ‘use extract’ to extract all of them).
As we are scraping, not the whole HTML but a small subset (the one in ‘book’) we need to put a dot at the start of the Xpath function. Remember: ‘//’ for the whole HTML response, ‘. //’ for a subset of that HTML we already extracted.
We have the title, now go the price. Right click the price and inspect it.
The text we want is inside a ‘p’ tag with the ‘price_color’ class inside a ‘div’ tag. Add this after the title:
price = (‘. //div/p[@class=”price_color”]/text()’). extract_first()
We go to any ‘div’, with a ‘p’ child that has a ‘price_color’ class, then we use ‘text()’ function to get the text. And then, we extract_first() our selection.
Let’s see if what we have. Print both the price and the title and run the spider.
print(title)
print(price)
Everything is working as planned. Let’s take the image URL too. Right-click the image, inspect it:
We don’t have an URL here but a partial one.
The ‘src’ attribute has the relative URL, not the whole URL. The ‘’ is missing. Well, we just need to add it. Add this at the bottom of your method.
image_url = art_urls[0] + (‘. //img[@class=”thumbnail”]/@src’). extract_first()
print(image_url)
We get the ‘img‘ tag with the class ‘thumbnail’, we get the relative URL with ‘src’ then we add the first (and only) start_url. Again, let’s print the result. Run the spider again.
Looking good! Open any of the URL and you’ll see the cover’s thumbnail.
Now let’s extract the URL so we can buy any book if we are interested.
The book URL is stored in the href of both the title and the thumbnail. Any of both will do.
book_url = art_urls[0] + (‘. //h3/a/@href’). extract_first()
print(book_url)
Run the spider again:
Click on any URL and you’ll go to that book website.
Now we are selecting all the fields we want, but we are not doing anything with it, right? We need to ‘yield’ (or ‘return’) them. For each book, we are going to return it’s title, price, image and book URL.
Remove all the prints and yield the items like a dictionary:
yield {
‘title’: title,
‘price’: price,
‘Image URL’: image_url,
‘Book URL’: book_url, }
Run the spider and look at the terminal:
Saving the data into a file
While it looks cool on the terminal, there is no use. Why don’t we store it into a file we can use later?
When we run our spider we have optional arguments. One of them is the name of the file you want to store. Run this.
scrapy crawl spider -o
Wait until it’s done… a new file has appeared! Double click it to open it.
All the information we saw on the terminal is now stored into a ‘’. Isn’t that cool? We can do the same with and files:
Conclusion
I know the first time is tricky, but you have learnt the basics of Scrapy. You know how to:
Create a Scrapy spider to navigate an URLA Scrapy project is structuredUse Xpath to extract the dataStore the data in, and files
I suggest you keep training. Look for an URL you want to scrape and try extracting a few fields as you did at the Beautiful Soup tutorial. The trick of Scrapy is learning how Xpath works.
But…do you remember that each book has an URL like this one?
Inside each item we scraped, there’s more information we can take. And we’ll do it in the second lesson of this series.
My Youtube tutorial videos
Final code on Github
Reach to me on Twitter
Your first Web Scraping script with Python and Beautiful Soup

A Minimalist End-to-End Scrapy Tutorial (Part I) | by Harry Wang

Systematic Web Scraping for BeginnersPhoto by Paweł Czerwiński on UnsplashPart I, Part II, Part III, Part IV, Part VWeb scraping is an important skill for data scientists. I have developed a number of ad hoc web scraping projects using Python, BeautifulSoup, and Scrapy in the past few years and read a few books and tons of online tutorials along the way. However, I have not found a simple beginner level tutorial that is end-to-end in the sense that covers all basic steps and concepts in a typical Scrapy web scraping project (therefore Minimalist in the title) — that’s why I am writing this and hope the code repo can serve as a template to help jumpstart your web scraping people ask: should I use BeautifulSoup or Scrapy? They are different things: BeautifulSoup is a library for parsing HTML and XML and Scrapy is a web scraping framework. You can use BeautifulSoup instead of Scrapy build-in selectors if you want but comparing BeautifulSoup to Scrapy is like comparing the Mac keyboard to the iMac or a better metaphor as stated in the official documentation “like comparing jinja2 to Django” if you know what they are:) — In short, you should learn Scrapy if you want to do serious and systematic web;DR, show me the code:In this tutorial series, I am going to cover the following steps:(This tutorial) Start a Scrapy project from scratch and develop a simple spider. One important thing is the use of Scrapy Shell for analyzing pages and debugging, which is one of the main reasons you should use Scrapy over BeautifulSoup. (Part II) Introduce Item and ItemLoader and explain why you want to use them (although they make your code seem more complicated at first). (Part III) Store the data to the database using ORM (SQLAlchemy) via Pipelines and show how to set up the most common One-to-Many and Many-to-Many relationships. (Part IV) Deploy the project to Scrapinghub (you have to pay for service such as scheduled crawling jobs) or set up your own servers completely free of charge by using the great open source project ScrapydWeb and Heroku. (Part V) I created a separate repo (Scrapy + Selenium) to show how to crawl dynamic web pages (such as a page that loads additional content via scrolling) and how to use proxy networks (ProxyMesh) to avoid getting prerequisites:Basic knowledge on Python (Python 3 for this tutorial), virtual environment, Homebrew, etc., see my other article for how to set up the environment: How to Setup Mac for Python DevelopmentBasic knowledge of Git and Github. I recommend the Pro Git knowledge of database and ORM, e. g., Introduction to Structured Query Language (SQL)’s get started! First, create a new folder, setup Python 3 virtual environment inside the folder, and install Scrapy. To make this step easy, I created a starter repo, which you can fork and clone (see Python3 virtual environment documentation if needed):$ git clone cd scrapy-tutorial-starter$ python3. 6 -m venv venv$ source venv/bin/activate$ pip install -r requirements. txtYour folder should look like the following and I assume we always work in the virtual environment. Note that we only have one package in the so scrapy startproject tutorial to create an empty scrapy project and your folder looks like:Two identical “tutorial” folders were created. We don’t need the first level “tutorial” folder — delete it and move the second level “tutorial” folder with its contents one-level up — I know this is confusing but that’s all you have to do with the folder structure. Now, your folder should look like:Don’t worry about the auto-generated files so far, we will come back to those files later. This tutorial is based on the official Scrapy tutorial. Therefore, the website we are going to crawl is, which is quite simple: there are pages of quotes with authors and tags:When you click the author, it goes to the author detail page with name, birthday, and, create a new file named “” in the “spider” folder with the following content:You just created a spider named “quotes”, which sends a request to and gets the response from the server. However, the spider does not do anything so far when parsing the response and simply outputs a string to the console. Let’s run this spider: scrapy crawl quotes, you should see the output like:Next, let’s analyze the response, i. e., the HTML page at using Scrapy Shell by running:$ scrapy shell 20:10:40 [] INFO: Spider opened2019-08-21 20:10:41 [] DEBUG: Crawled (404) (referer: None)2019-08-21 20:10:41 [] DEBUG: Crawled (200) (referer: None)[s] Available Scrapy objects:[s] scrapy scrapy module (contains quest, lector, etc)[s] crawler [s] item {}[s] request [s] response <200 >[s] settings [s] spider [s] Useful shortcuts:[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)[s] fetch(req) Fetch a quest and update local objects[s] shelp() Shell help (print this help)[s] view(response) View response in a browser>>>You can select elements using either Xpath selector or CSS selector and Chrome DevTools is often used to analyze the page (we won’t cover the selector details, please read the documents to learn how to use them):For example, you can test the selector and see the results in Scrapy Shell — assume we want to get the quote block shown above:You can either use Xpath (“//div[@class=’quote’]”)() (() shows the first selected element, use () to show all) or (“div ”)(). I bolded the quote text, author, and tags we want to get from this quote block:>>> (“//div[@class=’quote’]”)()’

Frequently Asked Questions about python spider tutorial