you need a lot of servers to get data within a reasonable time. You could consider hosting your scraper in the cloud and use a scalable Version of the Framework – like Scrapy Redis. For broader crawls
to run multiple spider instances to speed up crawls.
Frequently Asked Questions about There is a limit to the number of pages you can scrape from Amazon when using a single computer. If you’re scraping Amazon on a large scale
