Browser-based client-side scraping – Stack Overflow
No, you won’t be able to use the browser of your clients to scrape content from other websites using JavaScript because of a security measure called Same-origin policy.
There should be no way to circumvent this policy and that’s for a good reason. Imagine you could instruct the browser of your visitors to do anything on any website. That’s not something you want to happen automatically.
However, you could create a browser extension to do that. JavaScript browser extensions can be equipped with more privileges than regular JavaScript.
Adobe Flash has similar security features but I guess you could use Java (not JavaScript) to create a web-scraper that uses your user’s IP address. Then again, you probably don’t want to do that as Java plugins are considered insecure (and slow to load! ) and not all users will even have it installed.
So now back to your problem:
I need to scrape pages of an e-com site but several requests from the server would get me banned.
If the owner of that website doesn’t want you to use his service in that way, you probably shouldn’t do it. Otherwise you would risk legal implications (look here for details).
If you are on the “dark side of the law” and don’t care if that’s illegal or not, you could use something like to use IP adresses of real people.
Client-side web scraping with JavaScript using jQuery and …
by CodemzyWhen I was building my first open-source project, codeBadges, I thought it would be easy to get user profile data from all the main code learning websites. I was familiar with API calls and get requests. I thought I could just use jQuery to fetch the data from the various API’s and use name = ‘codemzy’; $(” + name, function(response) { var followers = llowers;});Well, that was easy. But it turns out that not every website has a public API that you can just grab the data you want from. 404: API not foundBut just because there is no public API doesn’t mean you need to give up! You can use web scraping to grab the data, with only a little extra ’s see how we can use client-side web scraping with an example, I will grab my user information from my public freeCodeCamp profile. But you can use these steps on any public HTML first step in scraping the data is to grab the full page html using a jQuery name = “codemzy”;$(” + name, function(response) { (response);});Awesome, the whole page source code just logged to the If you get an error at this stage along the lines of No ‘Access-Control-Allow-Origin’ header is present on the requested resource don’t fret. Scroll down to the Don’t Let CORS Stop You section of this was easy. Using JavaScript and jQuery, the above code requests a page from, like a browser would. And freeCodeCamp responds with the page. Instead of a browser running the code to display the page, we get the HTML that’s what web scraping is, extracting data from, the response is not exactly as neat as the data we get back from an … we have the data, in there we have the source code the information we need is in there, we just have to grab the data we need! We can search through the response to find the elements we ’s say we want to know how many challenges the user has completed, from the user profile response we got the time of writing, a camper’s completed challenges completed are organized in tables on the user profile. So to get the total number of challenges completed, we can count the number of way is to wrap the whole response in a jQuery object, so that we can use jQuery methods like () to get the data. // number of challenges completedvar challenges = $(response)(‘tbody tr’);This works fine — we get the right result. But its is not a good way to get the result we are after. Turning the response into a jQuery object actually loads the whole page, including all the external scripts, fonts and stylesheets from that page…Uh oh! We need a few bits of data. We really don’t need the page the load, and certainly not all the external resources that come with could strip out the script tags and then run the rest of the response through jQuery. To do this, we could use Regex to look for script patterns in the text and remove better still, why not use Regex to find what we are looking for in the first place? // number of challenges completedvar challenges = place(/
[\s|\S]*? <\/thead>/g)(/[ 1498]
just waiting to be points = (/
\[ ([\d]*? ) \]<\/h1>/)[1];In the above Regex pattern we match the h1 element we are looking for including the [] that surrounds the points, and group any number inside with ([\d]*? ). We get an array back, the first [0] element is the entire match and the second [1] is our group match (our points) is useful for matching all sorts of patterns in strings, and it is great for searching through our response to get the data we can use the same 3 step process to scrape profile data from a variety of websites:Use client-side JavaScriptUse jQuery to scrape the dataUse Regex to filter the data for the relevant informationUntil I hit a problem, Access DeniedDon’t Let CORS Stop You! CORS or Cross-Origin Resource Sharing, can be a real problem with client-side web security reasons, browsers restrict cross-origin HTTP requests initiated from within scripts. And because we are using client-side Javascript on the front end for web scraping, CORS errors can ’s an example trying to scrape profile data from CodeWars…var name = “codemzy”;$(” + name, function(response) { (response);});At the time of writing, running the above code gives you a CORS related there is noAccess-Control-Allow-Origin header from the place you’re scraping, you can run into bad news is, you need to run these sorts of requests server-side to get around this issue. Whaaaaaaaat, this is supposed to be client-side web scraping?! The good news is, thanks to lots of other wonderful developers that have run into the same issues, you don’t have to touch the back end aying firmly within our front end script, we can use cross-domain tools such as Any Origin, Whatever Origin, All Origins, crossorigin and probably a lot more. I have found that you often need to test a few of these to find the one that will work on the site you are trying to to our CodeWars example, we can send our request via a cross-domain tool to bypass the CORS name = “codemzy”;var url = ” + encodeURIComponent(“) + name + “&callback=? “;$(url, function(response) { (response);});And just like magic, we have our response.
Learn to code for free. freeCodeCamp’s open source curriculum has helped more than 40, 000 people get jobs as developers. Get started
artoo.js · The client-side scraping companion.
The client-side scraping companion
is a piece of JavaScript code meant to be run in your browser’s console to provide you with some scraping utilities.
This nice droid is loaded into the JavaScript context of any webpage through a handy bookmarklet you can instantly install by dropping the above icon onto your bookmark bar.
Disclaimer
Web security has widely improved in the last years and most websites prevent JavaScript code injection nowadays by relying on Content Security Policy headers.
But fear not, it is always possible to shunt them using browser extensions and/or proper configuration:
For chrome see the Disable Content-Security-Policy extension, for instance
For firefox, check this stackoverflow response explaining how to shunt CSP
Bootcamp
Now that you have installed let’s scrape the famous Hacker News in four painless steps:
Copy the following instruction.
(‘(3)’, {
title: {sel: ‘a’},
url: {sel: ‘a’, attr: ‘href’}}, vePrettyJson);
Go to Hacker News.
Open your JavaScript console and click the freshly created bookmarklet (the droid should greet you and tell you he is ready to roll).
Paste the instruction and hit enter.
That’s it. You’ve just scraped Hacker News front page and downloaded the data as a pretty-printed json file*.
* If you need a more thorough scraper, check this out.
Features
Scrape everything, everywhere: invoke artoo in the JavaScript context of any web page.
Loaded with helpers: Scrape data quick & easy with powerful methods such as
Data download: Make your browser download the scraped data with methods.
Spiders: Crawl pages through ajax and retrieve accumulated data with artoo’s spiders.
Content expansion: Expand pages’ content programmatically thanks to toExpand utilities.
Store: stash persistent data in the localStorage with artoo’s handy abstraction.
Sniffers: hook on XHR requests to retrieve circulating data with a variety of tools.
jQuery: jQuery is injected alongside artoo in the pages you visit so you can handle the DOM easily.
Custom bookmarklets: you can use artoo as a framework and easily create custom bookmarklets to execute your code.
User Interfaces: build parasitic user interfaces easily with a creative usage of Shadow DOM.
Philosophy
« Why on earth should I scrape on my browser? Isn’t this insane? »
Well, before quitting the present documentation and run back to your beloved spiders, you should pause for a minute or two and read the reasons why has made the choice of client-side scraping.
Usually, the scraping process occurs thusly: we find sites from which we need to retrieve data and we consequently build a program whose goal is to fetch those site’s html and parse it to get what we need.
The only problem with this process is that, nowadays, websites are not just plain html. We need cookies, we need authentication, we need JavaScript execution and a million other things to get proper data.
So, by the days, to cope with this harsh reality, our scraping programs became complex monsters being able to execute JavaScript, authenticate on websites and mimic human behaviour.
But, if you sit back and try to find other programs able to perform all those things, you’ll quickly come to this observation:
Aren’t we trying to rebuild web browsers?
So why shouldn’t we take advantage of this and start scraping within the cosy environment of web browsers? It has become really easy today to execute JavaScript in a
a browser’s console and this is exactly what is doing.
Using browsers as scraping platforms comes with a lot of advantages:
Fast coding: You can prototype your code live thanks to JavaScript browsers’ REPL and peruse the DOM with tools specifically built for web development.
No more authentication issues: No longer need to deploy clever solutions to enable your spiders to authenticate on the website you intent to scrape. You are already authenticated on your browser as a human being.
Tools for non-devs: You can easily design tools for non-dev people. One could easily build an application with a UI on top of Moreover, it gives you the possibility to create bookmarklets on the fly to execute your personnal scripts.
The intention here is not at all to say that classical scraping is obsolete but rather that client-side scraping is a possibility today and, what’s more, a useful one.
You’ll never find yourself crawling pages massively on a browser, but for most of your scraping tasks, client-side should enhance your productivity dramatically.
Contribution
Contributions are more than welcome. Feel free to submit any pull request as long as you added unit tests if relevant and passed them all.
To install the development environment, clone your fork and use the following commands:
# Install dependencies
npm install
# Testing
npm test
# Compiling dev & prod bookmarklets
gulp bookmarklets
# Running a test server hosting the concatenated file
npm start
# Running a server hosting the concatenated file
# Note that you’ll need some ssl keys (instructions to come… )
npm run
is being developed by Guillaume Plique @ SciencesPo – médialab.
Logo by Daniele Guido.
R2D2 ascii logo by Joan Stark aka jgs.
Under a MIT License.