you need a lot of servers to get data within a reasonable time. You could consider hosting your scraper in the cloud and use a scalable Version of the Framework – like Scrapy Redis. For broader crawls

to run multiple spider instances to speed up crawls.

Frequently Asked Questions about There is a limit to the number of pages you can scrape from Amazon when using a single computer. If you’re scraping Amazon on a large scale

Leave a Reply

Your email address will not be published. Required fields are marked *