Web Scraping HTML Tables with Python | by Syed Sadat Nazrul
Pokemon Database WebsiteStarting off, we will try scraping the online Pokemon Database () moving forward, we need to understand the structure of the website we wish to scrape. This can be done by clicking right-clicking the element we wish to scrape and then hitting “Inspect”. For our purpose, we will inspect the elements of the table, as illustrated below:Inspecting cell of HTML TableBased on the HTML codes, the data are stored in after
. This is the row information. Each row has a corresponding
or cell data will need requests for getting the HTML contents of the website and for parsing the relevant fields. Finally, we will store the data on a Pandas requestsimport as lhimport pandas as pdThe code below allows us to get the Pokemon stats data of the HTML ”#Create a handle, page, to handle the contents of the websitepage = (url)#Store the contents of the website under docdoc = omstring(ntent)#Parse data that are stored between
of HTMLtr_elements = (‘//tr’)For sanity check, ensure that all the rows have the same width. If not, we probably got something more than just the table. #Check the length of the first 12 rows[len(T) for T in tr_elements[:12]]OUTPUT: [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]Looks like all our rows have exactly 10 columns. This means all the data collected on tr_elements are from the, let’s parse the first row as our _elements = (‘//tr’)#Create empty listcol=[]i=0#For each row, store each first element (header) and an empty listfor t in tr_elements[0]: i+=1 name=t. text_content() print ‘%d:”%s”‘%(i, name) ((name, []))OUTPUT:1:”#”2:”Name”3:”Type”4:”Total”5:”HP”6:”Attack”7:”Defense”8:”Sp. Atk”9:”Sp. Def”10:”Speed”Each header is appended to a tuple along with an empty list. #Since out first row is the header, data is stored on the second row onwardsfor j in range(1, len(tr_elements)): #T is our j’th row T=tr_elements[j] #If row is not of size 10, the //tr data is not from our table if len(T)! =10: break #i is the index of our column i=0 #Iterate through each element of the row for t in erchildren(): data=t. text_content() #Check if row is empty if i>0: #Convert any numerical value to integers try: data=int(data) except: pass #Append the data to the empty list of the i’th column col[i][1](data) #Increment i for the next column i+=1Just to be sure, let’s check the length of each column. Ideally, they should all be the same. [len(C) for (title, C) in col]OUTPUT: [800, 800, 800, 800, 800, 800, 800, 800, 800, 800]Perfect! This shows that each of our 10 columns has exactly 800 we are ready to create the DataFrame_Dict={title:column for (title, column) in col}Frame(Dict)Looking at the top 5 cells on the ()There you have it! Now you have a Pandas DataFrame with all the information needed! This tutorial is a subset of a 3 part series:The series covers:Scraping a Pokemon WebsiteData AnalysisBuilding a GUI Pokedex
Scrape Tables From any website using Python – GeeksforGeeks
Scraping is a very essential skill for everyone to get data from any website. Scraping and parsing a table can be very tedious work if we use standard Beautiful soup parser to do so. Therefore, here we will be describing a library with the help of which any table can be scraped from any website easily. With this method you don’t even have to inspect element of a website, you only have to provide the URL of the website. That’s it and the work will be done within stallationYou can use pip to install this library: Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics. To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Coursepip install html-table-parser-python3Getting StartedStep 1: Import the necessary libraries required for the task# Library for opening url and creating
# requests
import quest
# pretty-print python data structures
from pprint import pprint
# for parsing all the tables present
# on the website
from import HTMLTableParser
# for converting the parsed data in a
# pandas dataframe
import pandas as pdStep 2: Defining a function to get contents of the website# Opens a website and read its
# binary contents (HTTP Response Body)
def url_get_contents(url):
# Opens a website and read its
#making request to the website
req = quest(url=url)
f = quest. urlopen(req)
#reading contents of the website
return ()Now, our function is ready so we have to specify the url of the website from which we need to parse Here we will be taking the example of website since it has many tables and will give you a better understanding. You can view the website here. Step 3: Parsing tables# defining the html contents of a URL.
xhtml = url_get_contents(‘Link’)(‘utf-8’)
# Defining the HTMLTableParser object
p = HTMLTableParser()
# feeding the html contents in the
# HTMLTableParser object
(xhtml)
# Now finally obtaining the data of
# the table required
pprint([1])Each row of the table is stored in an array. This can be converted into a pandas dataframe easily and can be used to perform any analysis. Complete Code:Python3import questfrom pprint import pprintfrom import HTMLTableParserimport pandas as pddef url_get_contents(url): req = quest(url=url) f = quest. urlopen(req) return ()xhtml = url_get_contents(‘/stockpricequote/refineries/relianceindustries/RI’)(‘utf-8’)p = HTMLTableParser()(xhtml)pprint([1])print(“\n\nPANDAS DATAFRAME\n”)print(Frame([1]))Output:
Extracting Data from HTML with BeautifulSoup – Pluralsight
IntroductionNowadays everyone is talking about data and how it is helping to learn hidden patterns and new insights. The right set of data can help a business to improve its marketing strategy and that can increase the overall sales. And let’s not forget the popular example in which a politician can know the public’s opinion before elections. Data is powerful, but it does not come for free. Gathering the right data is always expensive; think of surveys or marketing campaigns, etc. The internet is a pool of data and, with the right set of skills, one can use this data in a way to gain a lot of new information. You can always copy paste the data to your excel or CSV file but that is also time-consuming and expensive. Why not hire a software developer who can get the data into a readable format by writing some jiber-jabber? Yes, it is possible to extract data from Web and this “jibber-jabber” is called Web cording to Wikipedia, Web Scraping is:Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websitesBeautifulSoup is one popular library provided by Python to scrape data from the web. To get the best out of it, one needs only to have a basic knowledge of HTML, which is covered in the guide. Components of a WebpageIf you know the basic HTML, you can skip this basic syntax of any webpage is:1
2
3
4
5
6
7
8
My first Web Scraping with Beautiful soup
9
Let’s scrap the website using python.
10
11htmlEvery tag in HTML can have attribute information (i. e., class, id, href, and other useful information) that helps in identifying the element more information about basic HTML tags, check out for Scraping Any WebsiteTo scrape a website using Python, you need to perform these four basic steps:Sending an HTTP GET request to the URL of the webpage that you want to scrape, which will respond with HTML content. We can do this by using the Request library of Python. Fetching and parsing the data using Beautifulsoup and maintain the data in some data structure such as Dict or List. Analyzing the HTML tags and their attributes, such as class, id, and other HTML tag attributes. Also, identifying your HTML tags where your content lives. Outputting the data in any file format such as CSV, XLSX, JSON, etc. Understanding and Inspecting the DataNow that you know about basic HTML and its tags, you need to first do the inspection of the page which you want to scrape. Inspection is the most important job in web scraping; without knowing the structure of the webpage, it is very hard to get the needed information. To help with inspection, every browser like Google Chrome or Mozilla Firefox comes with a handy tool called developer this guide, we will be working with wikipedia to scrap some of its table data from the page List of countries by GDP (nominal). This page contains a Lists heading which contains three tables of countries sorted by their rank and its GDP value as per “International Monetary Fund”, “World Bank”, and “United Nations”. Note, that these three tables are enclosed in an outer know about any element that you wish to scrape, just right-click on that text and examine the tags and attributes of the into the CodeIn this guide, we will be learning how to do a simple web scraping using Python and stall the Essential Python Libraries1pip3 install requests beautifulsoup4 shellNote: If you are using Windows, use pip instead of pip3Importing the Essential LibrariesImport the “requests” library to fetch the page content and bs4 (Beautiful Soup) for parsing the HTML page content. 1from bs4 import BeautifulSoup
2import requestspythonCollecting and Parsing a WebpageIn the next step, we will make a GET request to the url and will create a parse Tree object(soup) with the help of BeautifulSoup and Python built-in “lxml” parser. 1# importing the libraries
2from bs4 import BeautifulSoup
3import requests
4
5url=”(nominal)”
6
7# Make a GET request to fetch the raw HTML content
8html_content = (url)
9
10# Parse the html content
11soup = BeautifulSoup(html_content, “lxml”)
12print(ettify()) # print the parsed data of htmlpythonWith our BeautifulSoup object i. e., soup we can move on and collect the required table going to the actual code, let’s first play with the soup object and print some basic information from it:Example 1:Let’s just first print the title of the will give an output as follows:1
2 print(“Inner Text: {}”())
3 print(“Title: {}”((“title”)))
4 print(“href: {}”((“href”)))pythonThis will output all the available links along with its mentioned attributes from the, let’s get back to the track and find our goal table. Analyzing the outer table, we can see that it has special attributes which include class as wikitable and has two tr tags inside you uncollapse the tr tag, you will find that the first tr tag is for the headings of all three tables and the next tr tag is for the table data for all three inner ‘s first get all three table headings:Note that we are removing the newlines and spaces from left and right of the text by using simple strings methods available in Python. 1gdp_table = (“table”, attrs={“class”: “wikitable”})
2gdp_table_data = (“tr”) # contains 2 rows
3
4# Get all the headings of Lists
5headings = []
6for td in gdp_table_data[0]. find_all(“td”):
7 # remove any newlines and extra spaces from left and right
8 ((‘\n’, ‘ ‘)())
10print(headings)pythonThis will give an output as:1[‘Per the International Monetary Fund (2018)’, ‘Per the World Bank (2017)’, ‘Per the United Nations (2017)’]Moving on to the second tr tag of the outer table, let’s get the content of all the three tables by iterating over each table and its rows.
1data = {}
2for table, heading in zip(gdp_table_data[1]. find_all(“table”), headings):
3 # Get headers of table i. e., Rank, Country, GDP.
4 t_headers = []
5 for th in nd_all(“th”):
6 # remove any newlines and extra spaces from left and right
7 ((‘\n’, ‘ ‘)())
8 # Get all the rows of table
9 table_data = []
10 for tr in (“tr”): # find all tr’s from table’s tbody
11 t_row = {}
12 # Each table row is stored in the form of
13 # t_row = {‘Rank’: ”, ‘Country/Territory’: ”, ‘GDP(US$million)’: ”}
14
15 # find all td’s(3) in tr and zip it with t_header
16 for td, th in zip(nd_all(“td”), t_headers):
17 t_row[th] = (‘\n’, ”)()
18 (t_row)
19
20 # Put the data for the table with his heading.
21 data[heading] = table_data
22
23print(data)pythonWriting Data to CSVNow that we have created our data structure, we can export it to a CSV file by just iterating over it. 1import csv
2
3for topic, table in ():
4 # Create csv file for each table
5 with open(f”{topic}”, ‘w’) as out_file:
6 # Each 3 table has headers as following
7 headers = [
8 “Country/Territory”,
9 “GDP(US$million)”,
10 “Rank”
11] # == t_headers
12 writer = csv. DictWriter(out_file, headers)
13 # write the header
14 writer. writeheader()
15 for row in table:
16 if row:
17 writer. writerow(row)pythonPutting It TogetherLet’s join all the above code snippets. Our complete code looks like this:1# importing the libraries
4import csv
5
7# Step 1: Sending a HTTP request to a URL
8url = “(nominal)”
9# Make a GET request to fetch the raw HTML content
10html_content = (url)
11
12
13# Step 2: Parse the html content
14soup = BeautifulSoup(html_content, “lxml”)
15# print(ettify()) # print the parsed data of html
16
17
18# Step 3: Analyze the HTML tag, where your content lives
19# Create a data dictionary to store the data.
20data = {}
21#Get the table having the class wikitable
22gdp_table = (“table”, attrs={“class”: “wikitable”})
23gdp_table_data = (“tr”) # contains 2 rows
24
25# Get all the headings of Lists
26headings = []
27for td in gdp_table_data[0]. find_all(“td”):
28 # remove any newlines and extra spaces from left and right
29 ((‘\n’, ‘ ‘)())
30
31# Get all the 3 tables contained in “gdp_table”
32for table, heading in zip(gdp_table_data[1]. find_all(“table”), headings):
33 # Get headers of table i. e., Rank, Country, GDP.
34 t_headers = []
35 for th in nd_all(“th”):
36 # remove any newlines and extra spaces from left and right
37 ((‘\n’, ‘ ‘)())
38
39 # Get all the rows of table
40 table_data = []
41 for tr in (“tr”): # find all tr’s from table’s tbody
42 t_row = {}
43 # Each table row is stored in the form of
44 # t_row = {‘Rank’: ”, ‘Country/Territory’: ”, ‘GDP(US$million)’: ”}
45
46 # find all td’s(3) in tr and zip it with t_header
47 for td, th in zip(nd_all(“td”), t_headers):
48 t_row[th] = (‘\n’, ”)()
49 (t_row)
50
51 # Put the data for the table with his heading.
52 data[heading] = table_data
53
54
55# Step 4: Export the data to csv
56″””
57For this example let’s create 3 seperate csv for
583 tables respectively
59″””
60for topic, table in ():
61 # Create csv file for each table
62 with open(f”{topic}”, ‘w’) as out_file:
63 # Each 3 table has headers as following
64 headers = [
65 “Country/Territory”,
66 “GDP(US$million)”,
67 “Rank”
68] # == t_headers
69 writer = csv. DictWriter(out_file, headers)
70 # write the header
71 writer. writeheader()
72 for row in table:
73 if row:
74 writer. writerow(row)pythonBEWARE -> Scraping rulesNow that you have a basic idea about scraping with Python, it is important to know the Legality of web scraping before starting scraping a website. Generally, if you are using scraped data for personal use and do not plan to republish that data, it may not cause any problems. Read the Terms of Use, Conditions of Use, and also the before scraping the website. You must follow the rules before scraping, otherwise, the website owner has every right to take legal action against nclusionThe above guide went through the process of how to scrape a Wikipedia page using Python3 and Beautiful Soup and finally exporting it to a CSV file. We have learned how to scrape a basic website and fetch all the useful data in just a couple of can further continue to expand the awesomeness of the art of scraping by jumping for new websites. Some good examples of data to scrape are:Customer reviews and product pagesBeautiful Soup is simple for small-scale web scraping. If you want to scrape webpages on a large scale, you can consider more advanced techniques like Scrapy and Selenium. Here are the some of my scraping guides:Hope you like this guide. If you have any queries regarding this topic, feel free to contact me at Links
Frequently Asked Questions about scrape table data from web page python
How do you scrape a table data from a website in Python?
To scrape a website using Python, you need to perform these four basic steps:Sending an HTTP GET request to the URL of the webpage that you want to scrape, which will respond with HTML content. … Fetching and parsing the data using Beautifulsoup and maintain the data in some data structure such as Dict or List.More items…•Dec 19, 2019
How do you scrape data from a website table?
In Google sheets, there is a great function, called Import Html which is able to scrape data from a table within an HTML page using a fix expression, =ImportHtml (URL, “table”, num). Step 1: Open a new Google Sheet, and enter the expression into a blank. A brief introduction of the formula will show up.Jun 30, 2020
Can Python pull data from website?
When scraping data from websites with Python, you’re often interested in particular parts of the page. By spending some time looking through the HTML document, you can identify tags with unique attributes that you can use to extract the data you need.