Scrapy Html

Selectors — Scrapy 2.5.1 documentation

When you’re scraping web pages, the most common task you need to perform is
to extract data from the HTML source. There are several libraries available to
achieve this, such as:
BeautifulSoup is a very popular web scraping library among Python
programmers which constructs a Python object based on the structure of the
HTML code and also deals with bad markup reasonably well, but it has one
drawback: it’s slow.
lxml is an XML parsing library (which also parses HTML) with a pythonic
API based on ElementTree. (lxml is not part of the Python
standard library. )
Scrapy comes with its own mechanism for extracting data. They’re called
selectors because they “select” certain parts of the HTML document specified
either by XPath or CSS expressions.
XPath is a language for selecting nodes in XML documents, which can also be
used with HTML. CSS is a language for applying styles to HTML documents. It
defines selectors to associate those styles with specific HTML elements.
Note
Scrapy Selectors is a thin wrapper around parsel library; the purpose of
this wrapper is to provide better integration with Scrapy Response objects.
parsel is a stand-alone web scraping library which can be used without
Scrapy. It uses lxml library under the hood, and implements an
easy API on top of lxml API. It means Scrapy selectors are very similar
in speed and parsing accuracy to lxml.
Using selectors¶
Constructing selectors¶
Response objects expose a Selector instance
on. selector attribute:
>>> (‘//span/text()’)()
‘good’
Querying responses using XPath and CSS is so common that responses include two
more shortcuts: () and ():
>>> (‘span::text’)()
Scrapy selectors are instances of Selector class
constructed by passing either TextResponse object or
markup as a string (in text argument).
Usually there is no need to construct Scrapy selectors manually:
response object is available in Spider callbacks, so in most cases
it is more convenient to use () and ()
shortcuts. By using lector or one of these shortcuts
you can also ensure the response body is parsed only once.
But if required, it is possible to use Selector directly.
Constructing from text:
>>> from lector import Selector
>>> body = ‘good‘
>>> Selector(text=body)(‘//span/text()’)()
Constructing from response – HtmlResponse is one of
TextResponse subclasses:
>>> from import HtmlResponse
>>> response = HtmlResponse(url=”, body=body)
>>> Selector(response=response)(‘//span/text()’)()
Selector automatically chooses the best parsing rules
(XML vs HTML) based on input type.
To explain how to use the selectors we’ll use the Scrapy shell (which
provides interactive testing) and an example page located in the Scrapy
documentation server:
For the sake of completeness, here’s its full HTML code:

Name: My image 1

Name: My image 2

Name: My image 3

Name: My image 4

Name: My image 5

First, let’s open the shell:
scrapy shell
Then, after the shell loads, you’ll have the response available as response
shell variable, and its attached selector in lector attribute.
Since we’re dealing with HTML, the selector will automatically use an HTML parser.
So, by looking at the HTML code of that
page, let’s construct an XPath for selecting the text inside the title tag:
>>> (‘//title/text()’)
[]
To actually extract the textual data, you must call the selector ()
or () methods, as follows:
>>> (‘//title/text()’)()
[‘Example website’]
‘Example website’
() always returns a single result; if there are several matches,
content of a first match is returned; if there are no matches, None
is returned. () returns a list with all results.
Notice that CSS selectors can select text or attribute nodes using CSS3
pseudo-elements:
>>> (‘title::text’)()
As you can see, () and () methods return a
SelectorList instance, which is a list of new
selectors. This API can be used for quickly selecting nested data:
>>> (‘img’)(‘@src’)()
[”,
”,
”]
If you want to extract only the first matched element, you can call the
selector () (or its alias. extract_first() commonly used in
previous Scrapy versions):
>>> (‘//div[@id=”images”]/a/text()’)()
‘Name: My image 1 ‘
It returns None if no element was found:
>>> (‘//div[@id=”not-exists”]/text()’)() is None
True
A default return value can be provided as an argument, to be used instead
of None:
>>> (‘//div[@id=”not-exists”]/text()’)(default=’not-found’)
‘not-found’
Instead of using e. g. ‘@src’ XPath it is possible to query for attributes
using property of a Selector:
>>> [[‘src’] for img in (‘img’)]
As a shortcut, is also available on SelectorList directly;
it returns attributes for the first matching element:
>>> (‘img’)[‘src’]
”
This is most useful when only a single result is expected, e. when selecting
by id, or selecting unique elements on a web page:
>>> (‘base’)[‘href’]
Now we’re going to get the base URL and some image links:
>>> (‘//base/@href’)()
>>> (‘base::attr(href)’)()
>>> (‘//a[contains(@href, “image”)]/@href’)()
>>> (‘a[href*=image]::attr(href)’)()
>>> (‘//a[contains(@href, “image”)]/img/@src’)()
>>> (‘a[href*=image] img::attr(src)’)()
Extensions to CSS Selectors¶
Per W3C standards, CSS selectors do not support selecting text nodes
or attribute values.
But selecting these is so essential in a web scraping context
that Scrapy (parsel) implements a couple of non-standard pseudo-elements:
to select text nodes, use::text
to select attribute values, use::attr(name) where name is the
name of the attribute that you want the value of
Warning
These pseudo-elements are Scrapy-/Parsel-specific.
They will most probably not work with other libraries like
lxml or PyQuery.
Examples:
title::text selects children text nodes of a descendant