Is it ok to scrape data from Google results? [closed] – Stack …
I’d like to fetch results from Google using curl to detect potential duplicate content.
Is there a high risk of being banned by Google?
asked Mar 26 ’14 at 10:07
0
Google disallows automated access in their TOS, so if you accept their terms you would break them.
That said, I know of no lawsuit from Google against a scraper.
Even Microsoft scraped Google, they powered their search engine Bing with it. They got caught in 2011 red handed:)
There are two options to scrape Google results:
1) Use their API
UPDATE 2020: Google has reprecated previous APIs (again) and has new
prices and new limits. Now
() you can
query up to 10k results per day at 1, 500 USD per month, more than that
is not permitted and the results are not what they display in normal
searches.
You can issue around 40 requests per hour You are limited to what
they give you, it’s not really useful if you want to track ranking
positions or what a real user would see. That’s something you are not
allowed to gather.
If you want a higher amount of API requests you need to pay.
60 requests per hour cost 2000 USD per year, more queries require a
custom deal.
2) Scrape the normal result pages
Here comes the tricky part. It is possible to scrape the normal result pages.
Google does not allow it.
If you scrape at a rate higher than 8 (updated from 15) keyword requests per hour you risk detection, higher than 10/h (updated from 20) will get you blocked from my experience.
By using multiple IPs you can up the rate, so with 100 IP addresses you can scrape up to 1000 requests per hour. (24k a day) (updated)
There is an open source search engine scraper written in PHP at It allows to reliable scrape Google, parses the results properly and manages IP addresses, delays, etc.
So if you can use PHP it’s a nice kickstart, otherwise the code will still be useful to learn how it is done.
3) Alternatively use a scraping service (updated)
Recently a customer of mine had a huge search engine scraping requirement but it was not ‘ongoing’, it’s more like one huge refresh per month.
In this case I could not find a self-made solution that’s ‘economic’.
I used the service at instead.
They also provide open source code and so far it’s running well (several thousand resultpages per hour during the refreshes)
The downside is that such a service means that your solution is “bound” to one professional supplier, the upside is that it was a lot cheaper than the other options I evaluated (and faster in our case)
One option to reduce the dependency on one company is to make two approaches at the same time. Using the scraping service as primary source of data and falling back to a proxy based solution like described at 2) when required.
answered Mar 28 ’14 at 2:35
JohnJohn6, 6733 gold badges47 silver badges49 bronze badges
12
Google will eventually block your IP when you exceed a certain amount of requests.
answered Mar 26 ’14 at 10:21
SeverinSeverin7, 99812 gold badges62 silver badges109 bronze badges
4
Google thrives on scraping websites of the if it was “so illegal” then even Google won’t survive. course other answers mention ways of mitigating IP blocks by Google. One more way to explore avoiding captcha could be scraping at random times (dint try) reover, I have a feeling, that if we provide novelty or some significant processing of data then it sounds fine at least to we are simply copying a website.. or hampering its business/brand in some it is bad and should be top of it you are a startup then no one will fight you as there is no benefit.. but if your entire premise is on scraping even when you are funded then you should think of more sophisticated ternative Google keeps releasing (or depricating) fields for its API so what you want to scrap now may be in roadmap of new Google API releases..
answered Jun 17 ’17 at 21:08
raghavraghav2072 silver badges5 bronze badges
Not the answer you’re looking for? Browse other questions tagged web-scraping or ask your own question.
Search engine scraping – Wikipedia
Search engine scraping is the process of harvesting URLs, descriptions, or other information from search engines such as Google, Bing, Yahoo, Petal or Sogou. This is a specific form of screen scraping or web scraping dedicated to search engines only.
Most commonly larger search engine optimization (SEO) providers depend on regularly scraping keywords from search engines, especially Google, Petal, Sogou to monitor the competitive position of their customers’ websites for relevant keywords or their indexing status.
Search engines like Google have implemented various forms of human detection to block any sort of automated access to their service, [1] in the intent of driving the users of scrapers towards buying their official APIs instead.
The process of entering a website and extracting data in an automated fashion is also often called “crawling”. Search engines like Google, Bing, Yahoo, Petal or Sogou get almost all their data from automated crawling bots.
Difficulties[edit]
Google is the by far largest search engine with most users in numbers as well as most revenue in creative advertisements, which makes Google the most important search engine to scrape for SEO related companies. [2]
Although Google does not take legal action against scraping, it uses a range of defensive methods that makes scraping their results a challenging task, even when the scraping tool is realistically spoofing a normal web browser:
Google is using a complex system of request rate limitation which can vary for each language, country, User-Agent as well as depending on the keywords or search parameters. The rate limitation can make it unpredictable when accessing a search engine automated as the behaviour patterns are not known to the outside developer or user.
Network and IP limitations are as well part of the scraping defense systems. Search engines can not easily be tricked by changing to another IP, while using proxies is a very important part in successful scraping. The diversity and abusive history of an IP is important as well.
Offending IPs and offending IP networks can easily be stored in a blacklist database to detect offenders much faster. The fact that most ISPs give dynamic IP addresses to customers requires that such automated bans be only temporary, to not block innocent users.
Behaviour based detection is the most difficult defense system. Search engines serve their pages to millions of users every day, this provides a large amount of behaviour information. A scraping script or bot is not behaving like a real user, aside from having non-typical access times, delays and session times the keywords being harvested might be related to each other or include unusual parameters. Google for example has a very sophisticated behaviour analyzation system, possibly using deep learning software to detect unusual patterns of access. It can detect unusual activity much faster than other search engines. [3]
HTML markup changes, depending on the methods used to harvest the content of a website even a small change in HTML data can render a scraping tool broken until it is updated.
General changes in detection systems. In the past years search engines have tightened their detection systems nearly month by month making it more and more difficult to reliable scrape as the developers need to experiment and adapt their code regularly. [4]
Detection[edit]
When search engine defense thinks an access might be automated the search engine can react differently.
The first layer of defense is a captcha page[5] where the user is prompted to verify they are a real person and not a bot or tool. Solving the captcha will create a cookie that permits access to the search engine again for a while. After about one day the captcha page is removed again.
The second layer of defense is a similar error page but without captcha, in such a case the user is completely blocked from using the search engine until the temporary block is lifted or the user changes their IP.
The third layer of defense is a long-term block of the entire network segment. Google has blocked large network blocks for months. This sort of block is likely triggered by an administrator and only happens if a scraping tool is sending a very high number of requests.
All these forms of detection may also happen to a normal user, especially users sharing the same IP address or network class (IPV4 ranges as well as IPv6 ranges).
Methods of scraping Google, Bing, Yahoo, Petal or Sogou[edit]
To scrape a search engine successfully the two major factors are time and amount.
The more keywords a user needs to scrape and the smaller the time for the job the more difficult scraping will be and the more developed a scraping script or tool needs to be.
Scraping scripts need to overcome a few technical challenges:[6]
IP rotation using Proxies (proxies should be unshared and not listed in blacklists)
Proper time management, time between keyword changes, pagination as well as correctly placed delays Effective longterm scraping rates can vary from only 3–5 requests (keywords or pages) per hour up to 100 and more per hour for each IP address / Proxy in use. The quality of IPs, methods of scraping, keywords requested and language/country requested can greatly affect the possible maximum rate.
Correct handling of URL parameters, cookies as well as HTTP headers to emulate a user with a typical browser[7]
HTML DOM parsing (extracting URLs, descriptions, ranking position, sitelinks and other relevant data from the HTML code)
Error handling, automated reaction on captcha or block pages and other unusual responses[8]
Captcha definition explained as mentioned above by[9]
An example of an open source scraping software which makes use of the above mentioned techniques is GoogleScraper. [7] This framework controls browsers over the DevTools Protocol and makes it hard for Google to detect that the browser is automated.
Programming languages[edit]
When developing a scraper for a search engine almost any programming language can be used. Although, depending on performance requirements, some languages will be favorable.
PHP is a commonly used language to write scraping scripts for websites or backend services, since it has powerful capabilities built-in (DOM parsers, libcURL); however, its memory usage is typically 10 times the factor of a similar C/C++ code. Ruby on Rails as well as Python are also frequently used to automated scraping jobs. For highest performance, C++ DOM parsers should be considered.
Additionally, bash scripting can be used together with cURL as a command line tool to scrape a search engine.
Tools and scripts[edit]
When developing a search engine scraper there are several existing tools and libraries available that can either be used, extended or just analyzed to learn from.
iMacros – A free browser automation toolkit that can be used for very small volume scraping from within a users browser [10]
cURL – a command line browser for automation and testing as well as a powerful open source HTTP interaction library available for a large range of programming languages. [11]
google-search – A Go package to scrape Google. [12]
SEO Tools Kit – Free Online Tools, Duckduckgo, Baidu, Petal, Sogou) by using proxies (socks4/5, proxy). The tool includes asynchronous networking support and is able to control real browsers to mitigate detection. [13]
se-scraper – Successor of SEO Tools Kit. Scrape search engines concurrently with different proxies. [14]
Legal[edit]
When scraping websites and services the legal part is often a big concern for companies, for web scraping it greatly depends on the country a scraping user/company is from as well as which data or website is being scraped. With many different court rulings all over the world. [15][16][17]
However, when it comes to scraping search engines the situation is different, search engines usually do not list intellectual property as they just repeat or summarize information they scraped from other websites.
The largest public known incident of a search engine being scraped happened in 2011 when Microsoft was caught scraping unknown keywords from Google for their own, rather new Bing service, [18] but even this incident did not result in a court case.
One possible reason might be that search engines like Google, Petal, Sogou are getting almost all their data by scraping millions of public reachable websites, also without reading and accepting those terms.
See also[edit]
Comparison of HTML parsers
References[edit]
^ “Automated queries – Search Console Help”. Retrieved 2017-04-02.
^ “Google Still World’s Most Popular Search Engine By Far, But Share Of Unique Searchers Dips Slightly”. 11 February 2013.
^ “Does Google know that I am using Tor Browser? “.
^ “Google Groups”.
^ “My computer is sending automated queries – reCAPTCHA Help”. Retrieved 2017-04-02.
^ “Scraping Google Ranks for Fun and Profit”.
^ a b “Python3 framework GoogleScraper”. scrapeulous.
^ Deniel Iblika (3 January 2018). “De Online Marketing Diensten van DoubleSmart”. DoubleSmart (in Dutch). Diensten. Retrieved 16 January 2019.
^ Jan Janssen (26 September 2019). “Online Marketing Services van SEO SNEL”. SEO SNEL (in Dutch). Services. Retrieved 26 September 2019.
^ “iMacros to extract google results”. Retrieved 2017-04-04.
^ “libcurl – the multiprotocol file transfer library”.
^ “A Go package to scrape Google” – via GitHub.
^ “Free online SEO Tools (like Google, Yandex, Bing, Duckduckgo,… ). Including asynchronous networking support. : NikolaiT/SEO Tools Kit”. 15 January 2019 – via GitHub.
^ Tschacher, Nikolai (2020-11-17), NikolaiT/se-scraper, retrieved 2020-11-19
^ “Is Web Scraping Legal? “. Icreon (blog).
^ “Appeals court reverses hacker/troll “weev” conviction and sentence [Updated]”.
^ “Can Scraping Non-Infringing Content Become Copyright Infringement… Because Of How Scrapers Work? “.
^ Singel, Ryan. “Google Catches Bing Copying; Microsoft Says ‘So What? ‘”. Wired.
External links[edit]
Scrapy Open source python framework, not dedicated to search engine scraping but regularly used as base and with a large number of users.
Compunect scraping sourcecode – A range of well known open source PHP scraping scripts including a regularly maintained Google Search scraper for scraping advertisements and organic resultpages.
Justone free scraping scripts – Information about Google scraping as well as open source PHP scripts (last updated mid 2016)
rvices source code – Python and PHP open source classes for a 3rd party scraping API. (updated January 2017, free for private use)
PHP Simpledom A widespread open source PHP DOM parser to interpret HTML code into variables.
SerpApi Third party service based in the United States allowing you to scrape search engines legally.
How to Scrape Google Without Coding – ScrapeHero Cloud
This tutorial will show you how to scrape Google data for free using the ScrapeHero Cloud. Using these crawlers we will be scraping Google Search Results Page, Google Maps, and Google Reviews.
Here are the steps to scrape Google
Create the ScrapeHero Cloud account
Select the Google crawler you would like to run – Google Search Result Scraper, Google Maps Scraper, or Google Reviews Scraper
Enter the list of input URLs
Run the scraper and download the data
The ScrapeHero Cloud has pre-built scrapers that can Scrape Job data, Scrape Real Estate Data, Scrape Social Media and more. Web scraping using ScrapeHero Cloud is easy as the crawlers are cloud-based and you need not worry about selecting the fields to be scraped nor download any software. The scraper and the data can be accessed from any browser at any time. You can also get the data delivered directly to your Dropbox.
If you don’t like or want to code, ScrapeHero Cloud is just right for you!
Skip the hassle of installing software, programming and maintaining the code. Download this data using ScrapeHero cloud within seconds.
Get Started for Free
Create a ScrapeHero Cloud Account
Before using a crawler in ScrapeHero Cloud, an account must be created. To sign up, go to – and create an account with your email address.
Each account lets you test a crawler by allowing you to scrape 25 pages for free before subscribing. Below we have provided a detailed explanation on how to use the different Google crawlers that are available on ScrapeHero Cloud.
How to Scrape Google Search Results Page
Since Google does not provide an API, it is difficult to gather Google search results data without purchasing expensive tools. We can scrape Google SERP data using the Google Search Results Scraper in the ScrapeHero Cloud. The ScrapeHero Cloud allows you to scrape Google search result pages for a variety of search terms in a fast and cost-effective manner.
Using the ScrapeHero Cloud, you can scrape Google search results to gather details from Google Knowledge Graph, monitor organic and paid search results, gather news articles, and more within a few clicks.
Here is what we will scrape from Google Search Results Page
Search Rank
Results Type
Title
URL
Breadcrumb
Description
Published Date
Search Keyword
Infobox (Knowledge Graph)
This crawler accepts input based on a search query. Here is an example – Top universities in UK
You may add as many queries as required, as long as a new line separates each one. After you have input all the search queries, enter the number of pages to scrape. If you keep this section blank the crawler will only download the results from the first page of Google search results.
You can start scraping the data by clicking on ‘Gather Data’. When you click on ‘View Data’ you have the option to choose either the Infobox or Search Results.
The crawler gives you the option to download the Search Results and the Infobox (Google Knowledge Graph) as two separate datasets.
Here is a sample of what the scraped Google Search Results data will look like:
How to scrape Google Maps
Google Maps allows users to search for businesses in any zip code, county, city, state, or country using specific keywords. The Google Maps Search Results crawler allows you to gather business information from Google Maps by entering a keyword and location combination.
While we can use Google Maps to find businesses manually, this would be an arduous process. Using ScrapeHero Cloud automates the process of extracting data from Google maps on a large scale and can help generate sales leads.
Here is what we will scrape from Google Maps
Business Name
Address
Phone Number
Website
Rating
Reviews
Category
Status
GeoCoordinates
Place ID
Review URL
Timings
Log in to your ScrapeHero Cloud account and add the Google Maps Scraper.
Next, provide the input for the crawler. Here is an example – restaurants in Boston
You can add as many keywords as you would like with each one separated by a new line. Once you provide a list of inputs to the crawler and start the crawler, it will take a few minutes to scrape all the results from the Google Maps results page.
In addition, you also have the option to schedule the scraper to run on a regular interval, allowing you to check for new businesses in an area using Google Maps.
Here is a sample of what the scraped Google Maps data will look like:
How to scrape Google Reviews
Google Reviews improves local search ranking, trust and credibility with consumers, and can influence consumer decisions. A user can search for a business and find reviews and ratings using Google Reviews.
The Google Reviews Scraper by ScrapeHero Cloud allows you to gather information from Google Reviews based on Google review URLs or Place IDs.
Here is what we will scrape from Google Reviews
Aggregated Rating
Total Reviews
Ratings
Author
Reviews Images
Posted Date
After creating an account in ScrapeHero Cloud we will obtain a Google review URL based on a query in the Google search bar. Here is an example for Eiffel Tower –
You will see the link to the Google reviews in the Infobox.
The scraper also accepts Google Place ID as inputs. You can use the Google Maps crawler mentioned above to gather Place IDs and use those as input for the Google Review Scraper.
Once you provide the review URLs/Place IDs you can get all the scraped review data in minutes. The crawler can scrape reviews using filters such as the most relevant, newest, highest, and lowest rating.
You can run the scraper on a schedule to keep getting new and updated Google reviews. Here is a sample of what the scraped Google Review data will look like:
If you need to scrape Google with better location-specific results or need more data fields and attributes, ScrapeHero can create a custom plan for you and help you get started.
How to Scrape Google Search Results without getting banned?
ScrapeHero Cloud can scrape Google search results without getting blocked. It has been designed to avoid IP bans and CAPTCHA to ensure that users can scrape 1000+ search queries at a time. You need not worry about getting blocked due to scraping or rotating proxies, ScrapeHero Cloud helps you scrape Google without getting blocked. Just provide your inputs to the crawler and wait for the crawler to complete running.
The internet is full of information and Google’s search engine is remarkably the best when it comes to returning search results. Not only can web scraping Google show a company how high their website page appears on a Google results page, but it can also give a glimpse of how many keywords their website is using on any given page. Knowing how to utilize SEO will keep your business highly competitive and scraping Google results is a tactical way to gain an understanding of those SEO practices.
When marketing is a huge factor in introducing a brand to the public, a tool like ScrapeHero Cloud will help collect data to know how competitors are advertising products. The more data you have, the better you are able to market to a target audience and relate to potential customers. Utilizing the extraction abilities of a Google scraper is a fast, effective way to understand customers and cultivate unique marketing tactics.
We can help with your data or automation needs
Turn the Internet into meaningful, structured and usable data
Frequently Asked Questions about scrape google results
Is it legal to scrape Google results?
Although Google does not take legal action against scraping, it uses a range of defensive methods that makes scraping their results a challenging task, even when the scraping tool is realistically spoofing a normal web browser: … Network and IP limitations are as well part of the scraping defense systems.
How do I scrape Google search results?
Here are the steps to scrape GoogleCreate the ScrapeHero Cloud account.Select the Google crawler you would like to run – Google Search Result Scraper, Google Maps Scraper, or Google Reviews Scraper.Enter the list of input URLs.Run the scraper and download the data.Sep 8, 2020