WP Content Crawler – WordPress Crawler Plugin For Web
WP Content Crawler plugin lets you grab content & posts automatically from any of your favorite sites. Also, you can save custom fields, taxonomies, set cookies etc. with the powerful crawling tool.
Web Crawler is the most advanced WordPress plugin to grab content & post automatically from any web site for creating a personal website.
With this plugin, you can make a collection of various theme, plugins, apps & images. Also, collect various products from shopping sites, and get affiliate programs to make money.
Additionally, manually you can crawl the website content with several features such as cookies, quick fixes, proxies, warnings, duplicate post checking etc.
Key Features Of WordPress Crawler Plugin For Web
Visual Selector (Visual Inspector)
You can click to an element for finding its CSS selector and get its alternative way according to the requirement without leaving the admin panel.
Recrawl (Update) Posts
Web content crawler allows you to recrawl posts, contents automatically to keep them updated all the time. You can control scheduling and set the limit for post updates, set update interval, and neglect old posts.
Save Every Post Detail
You can save the custom post and their details using post meta and taxonomy options. Save every information of the website like title, excerpt, content, tags, slug (permalink), date, featured image, post images, status, content templates etc.
Find and Replace Anything
In web crawler WordPress plugin you can use plain text or regular expressions to find and replace content. Also, it allows you can create and modify the HTML elements and image URLs of the page.
Password Protection
The plugin allows you to protect information from unauthorized users by setting a password for the posts. Also, the users having the password can view the post.
Email Notifications
You can set CSS selector for empty category and post pages. Using those selectors, you can get an email notification for empty value.
Create Gallery
WP content crawler plugin has several templates to create an image gallery and save the images as WooCommerce gallery by just checking one checkbox.
Cookies
It allows you to elegantly inform users by attaching cookies to each request such as session cookies for crawling the targeted site.
Comprehensive Dashboard
With this plugin, you get a comprehensive dashboard that provides several options to see background details like-
Site Activity
Number of posts crawled
Number of updated posts
Last crawled & added URLs
Last and next run of CRON events etc.
Option Box
Define many options like calculation, template, and JSON parsing settings for finding the values by a CSS selector.
Automatic Translation
In this plugin, use the artificial intelligence of Google Cloud Translation API or Microsoft Translator Text API to translate the posts automatically.
Some Other Prominent Features-
It supports the latest versions of PHP.
Add & activate unlimited numbers of sites.
Enable/disable automatic crawling for a specific website.
Add URLs to the database using the manual crawling tool.
Show the details of the error so that you can fix it right away.
You can convert the encoding by checking a single checkbox and handle different character encodings.
Without waiting for page reloading you can you can save the settings quickly with quicksave button.
You can import and export site settings easily by copy and paste the plugin code.
With templates, you can save media file for defining title, description, caption, and alt texts for using any shortcode.
You May Also Like:
WordPress Facebook Feed Plugin
Product Catalog WordPress Plugin
Learning Management System WordPress Plugin
Automatic WordPress Scraper and Content Crawler Plugin …
Scrapes
The most advanced WordPress scraper and content crawler plugin to scrape content from any website automatically with visual selector.
Try live demo
View pricing
Get your content ready in minutes.
The easy to use interface of our automatic WordPress content crawler plugin is designed for the best user experience. Set up easily in a few steps and leave it in the background to get content from source sites.
Enter target URL to scrape
Inspect with visual selector
Run WordPress autoblog plugin
One WordPress autoblog plugin for all sources.
What Scrapes can do is completely limited to your imagination. Use the default settings for popular sources or create new tasks from scratch and crawl through almost all websites or use it as a RSS aggregator plugin for WordPress.
Usecase ideas.
News
Currencies
Weather forecasts
Live sport scores
Blogs & Magazines
Articles
Bulletin boards
Social profiles
Design inspirations
Images & Galleries
Videos
Movies & Celebrities
Music & Playlists
Apps & Games
Books
Themes & Plugins
Products
Freelance jobs
Syllabi
Restaurants
Recipes
Theaters
Airports & Flights
Transportation hours
Hotels
+ more
All unique and powerful features.
We have identified all the features that an automatic WordPress scraper plugin must have, and developed the state of the art Scrapes to meet all needs from the simplest to most advanced.
Instant control with detailed dashboard.
View all of your tasks on a single screen and easily manage them as you wish.
Supports all WordPress fields.
Scrapes automatically fills all fields those are supported, as if you had manually filled them out. Results of automatic WordPress content crawler plugin is no different from native one.
Post type
Taxonomies
Title
Content
Excerpt
Custom fields
Discussion
Author
Status
Date
Image
Visual selector.
You do not need any programming skills to make the necessary settings. All you need is to match the parts you need on the target page and the corresponding WordPress field with the visual selector.
Auto detect.
Content, featured image, next page etc. parts are matched with the corresponding fields automatically with automatic WordPress scraper plugin from the suitable source sites, so the most accurate section is provided for scraping with minimum effort.
RSS aggregator plugin for WordPress.
Get up to date subjects from feeds of websites which are not suitable for plain HTML scraping. Meet both your WordPress auto post plugin and RSS aggregator plugin for WordPress needs.
Advanced content customization.
You do not have to scrape the content with automatic WordPress content crawler plugin as is. Improve the layout automatically according to your needs before adding it to your own site.
Templates.
Specify the order of scraped data you want to appear on your site with the template option and personalize the layout of your post. All the options you can use are available as a template tag in the interface.
Usecase ideas: Add the origin site’s address which you get with WordPress auto post plugin to the end of the body, combine the different parts, and edit the layout using visual composer tags.
Find & Replace.
Remove specific words and phrases from the scraped text with the regular expression-supported find & replace feature, or replace them with your own words. Moreover, you can add an unlimited number of rules to run respectively.
Usecase ideas: Remove or replace links in the page, publish only a few words or sentences, remove ads with automatic WordPress content crawler plugin.
Math calculation.
Perform mathematical operations such as addition, subtraction, multiplication, and division on scraped data and obtain a new number. Combine numbers in different custom fields and create new formulas via automatic WordPress scraper plugin.
Usecase ideas: Include your own commission at the price before publishing a WooCommerce product.
Language translation.
Automatically translate content scraped in a specific language to one of dozens of different language options to get completely new content with Bing Microsoft Translator, DeepL Translator, Google Translate or Yandex Translate service.
Usecase ideas: Translate an English website published in another country to your local language with WordPress autoblog plugin.
Content spinner.
You can benefit from integrated ready to use 3rd party spinner services, or use the existing spinner plugins to alter scraped content.
Usecase ideas: Use The Best Spinner service with our RSS aggregator plugin for WordPress to differentiate the content.
Filters.
Filter the content in the source site to match the rules you specify. So, just make sure that the content is successfully passed through these filters is added to your own site. Moreover, you can add an unlimited number of rules.
Usecase ideas: Through our WordPress auto post plugin, pass posts with a specific word in title, just scrape content after a certain date, skip posts without a featured image, just scrape products under a certain price.
Integrated with themes and plugins.
If you just need more than automatic WordPress scraper plugin, it is for you. Running in compatibility with other themes and plugins is just one of the features that make it strong.
Works with WooCommerce.
From the online stores, you can use the custom post type and custom fields support in your WooCommerce store to collect content in the form of products. Earn revenue by selling products either in your own store or redirecting them to the target site through an affiliate program.
Product type
Regular price
Sale price
Product URL
Button text
SKU
Manage stock
Stock
Backorders
Stock status
Purchase note
Visibility
Featured
Gallery
Superfast and high quality support.
You receive top notch support with dedicated team of fastest professionals across the world.
Trusted by the community.
55. 000+ Satisfied customers can’t be wrong.
No additional costs or hidden fees, one time payment only.
Thanks to our transparent price policy and automatic activation system, you can get started with the download link and license code to be sent to your e-mail address after you complete the payment process.
Standard
$29/ per license
$58
50% OFF Limited time offer!
Lifetime version updates
Lifetime technical support
Instant download and use
OFFER GET EXTRA 50% OFF
On 5+ Scrapes purchases.
OFFER GET EXTRA 60% OFF
On 10+ Scrapes purchases.
OFFER GET EXTRA 75% OFF
On 30+ Scrapes purchases.
F. A. Q
We have answered and gathered all the questions you might have about Scrapes in detail.
Scrapes is designed to provide maximum performance even in the minimum system configuration provided by shared hosting. However, issues such as the hardware specification of your server, the connection speed of your server and the source site, some limitations defined on your server, how many tasks you are running at the same time etc. are factors that affect performance. The minimum requirements are listed below.
PHP 5. 2. 4+
dom, mbstring, iconv, json and simplexml extensions for PHP
Configurable maximum execution time for PHP
WordPress 3. 5+
Google Chrome, Firefox, Safari or Microsoft Edge for admin panel
You can already copy the content of any site to your own WordPress site manually without any tools. Scrapes is simply an automatic WordPress scraper plugin that allows you to automate this process. In other words, this question is not related to Scrapes, and there is no clear answer to “Yes” or “No”. Actually the answer differs according to the source site and your intended use of the content.
Some aim to reach more users by sharing their content with content sharing systems such as RSS (Rich Site Summary) or API (Application Programming Interface).
Some allow you to publish some of their content, ask you to give them the backlink to view the rest and target more hits.
Some in exchange for publishing their products on your site, pay you under their affiliate marketing programs and aim to sell more.
Some allow you to publish their content except commercial use.
Some refer to extra protection methods in order not to scrape their contents.
Although it is entirely your responsibility to copy the content of any website, our recommendations for not doing something illegal and avoiding Google penalties of duplicate content are as follows.
In necessary situation, you can contact the owner and request permission to publish the content in your site.
Using an SEO plug-in that allows you to define a Canonical URL, you can specify which source is used.
Change content to a different language with the translate feature, or spin and retrieve a new content with content spinner services.
With the template feature, you can add credit information or the backlink directed to the origin URL to the end of the content in WordPress auto post plugin mode for example.
You can publish a portion of the content by shortening it with Find & Replace feature instead of publishing the entire content.
You can extract content in “Draft” status and publish it after making changes.
Scrapes is designed to run compatible with other WordPress plugins and themes however some themes and plugins may require some complex string type values called array like a:1:{i:20041001103319;s:4:”test”;} which is not supported by our product.
In order to check your theme or plugin needs this type of strings, you can install free “Post Meta Inspector” plugin from. You will see an additional meta box on your posts / custom post types at the end of page like this screenshot after the installation. If every meta field has plain text values then you can easily use Scrapes to fill every data you need from the target website, otherwise you will need some manual edits after the scrape.
If the “Remember me” option is checked when logging in, the login information is stored in the browser as a cookie in encrypted form. You can extract content from resource sites which require logging, or using Cloudflare by defining these cookie values to Scrapes.
Scrapes can extract content from almost all sites and RSS feeds, except as noted type of sites below. You can find out whether the target you are planning to scrape is appropriate, by trying out the live demo of our product or by contacting us.
The content should be loaded when the page is opened. JavaScript / Ajax technologies and post-loaded page content are not supported.
On the listing page, there should be a “Pagination” or “Next page” button that redirects to the next page. The content pages which are loading dynamic content without address change like “load more” button pressing or using infinite scrolls are partially supported.
On the listing page, there should be links in a given order and directed to the detail pages. Listing pages with links that are not positioned in a specific order, opening pages in a popup instead of link redirection, or loading content after page load action with JavaScript / Ajax technologies are not supported.
The detail pages that are redirected from the listing page should have the same HTML page structure and must be a valid HTML. Non-valid detail pages with different HTML page structure may not be supported.
RSS and Atom feeds with non-valid XML structure may not be supported by our RSS aggregator plugin for WordPress.
There is no any limitation in our WordPress autoblog plugin to add an unlimited number of tasks, However each task may consume 10 – 30 MB of RAM during the process and they may slow down or not work properly if you have too many tasks running at the same time considering your server resources.
In order to reduce performance loss and prevent IP bans we suggest you to reduce your tasks or edit them to run in different timespans.
It can create “Simple”, “External” and “Virtual” WooCommerce products automatically from the target you want but “Grouped”, “Variable” and “Downloadable” product types and product attributes are not fully supported currently.
What Scrapes can do is completely limited to your imagination. You can create a site that will have rich content and get click, generate ad revenue, or open a store and earn commission revenue with affiliate marketing programs.
Scraper – Automatic Content Crawler Plugin for WordPress
Scraper is an automatic plugin that copies content and posts automatically from any web site. With tons of useful and unique features, Scraper WordPress plugin takes content creating process to another level.
Design your own scraping method with visual editor, Scraper will automatically fetch and post 7/24 for you.
Tutorials
We are creating step by step tutorials for plugin. Please let us know if you have any questions, we’ll be adding all details on tutorials.
Product
Troubleshoot
Settings
Other
Structures
Features
Scrape Any Website
With Xpath and regex methods, it’s possible to scrape any website!
Get Attributes
Scraper can parse element’s attributes, that means you could get links, image sources, video sources.
Set Feature Image
The plugin can extract any image to set as a featured image.
Duplicate Title Skip
Scraper verifies that there is no previous post with the same title.
Fix Encoding
The plugin automatically fix encoding errors and post it after fixing it.
Language Translation
Scraper automatically detects content and translate it to any language.
Support Categories
You could post your new content to any category that you’ve created.
Regex (JSON) Parser
You could parse an information that contains JSON objects.
Gallery Images
Scraper can parse any image sequence and create gallery with that.
Search & Replace
You could transform your content with search & replace feature.
Content Template with Variables
You could generate your unique content template with transform feature.
WooCommerce Products
Creating woocommerce products with Scraper is really easy. Scraper supports all tags of woocommerce.
Math Functions
You could sum, subtract or divide any number with math functions. It can be useful on price calculations.
Embed Any Post
Extract Youtube, Instagram or Vimeo ID from website and use it with embed transform.
Schedule Tasks
With different intervals you could schedule your tasks.
Post Update
You could select a post to update with scraping.
Link Strip
You could strip links from original post content.
Proxy Support
You could use proxies on scraping.
Cookie Support
Scraper supports cookie, you could define cookie for any task.
Post Status
You could set your post status to draft or publish.
Community Templates
There are hundreds of template on Scraper’s library. You could enable any of them easily.
F. A. Q.
+ Is there any feature to translate content?
Yes, it translates any content with Google Translate API. It supports all languages that supported by Google.
+ Is it possible to set featured image?
Yes, it’s possible. You could set any image element as a featured image.
+ Is it possible to create image galleries?
Yes, it’s possible. Scraper can detect image sequences and define them as gallery. It can also downloads image to your server.
+ Is it possible to create WooCommerce product?
Yes, we support all WooCommerce product custom fields and post types.
+ Is it possible to update prices on product?
Yes, with transform function, you could apply some regex expressions or mathematical functions for numbers. With this way, you could update numbers with yours or change the values with math functions.
+ Is it possible to scrape meta tags?
Yes, that’s possible. You could scrape HTML tags, meta tags, attributes or even JSON parts of page with visual editor.
+ Is it possible to JSON values on page?
Yes, that’s possible. You could write your own regex expression to extract data from JSON content.
+ Can i update test my task model before i create it?
Yes, in preview results section, you could test your model before you purchase or use plugin. It provides same results with WordPress plugin.
+ Can i use proxy for my content?
Yes, you could set proxy IP and port on settings button which is located beside fetch button.
+ Can i scrape all category with next page?
Yes, setting next page element is enough. It will automatically scrape all products or post items in a category page.
+ Can i get support for my task?
Yes, our support team will be asisting on your task or any issue that you experience on plugin.
+ Can i get refund?
Yes, if the plugin is not working as expected. Please reachout to use via before raising any refund request.
See Plugin’s Documentation
Visit Frequenly Asked Questions
Next Release
Changelog
04/30/2021 (v2. 0. 4)
– Bug: Fixed issue while adding more than 26 fields.
– Bug: Fixed Scheduled tasks get stuck in queue
– Bug: Fixed key name for new data field
– Added Cron URLs that can be run directly via server’s Cron Job
12/16/2020 (v2. 3)
– Bug: Fixed Duplicate Image download
– Bug: Fixed XML/RSS feed scrapping
– Feature: Added Task URLs that can be run directly via server’s Cron Job
– Feature: Added option to export output to excel directly without posting it to WP
– Added: Ready to use templates
– Removed Unnecessary Files & Folders
05/19/2020 (v2. 2)
– Bug: Author not set when posts imported via scheduled tasks
05/14/2020 (v2. 1)
04/27/2020 (v2. 0)
– Introducing Scraper Pro (for ajax scraping)
– Updated Documentation link
– Feature: Added support for `Alt` & `Title` for featured image
– Feature: Green Highlighter for Feed link selector
– Bug: Fixed warning issues
– Bug: Fixed Incorrect XML / RSS parsing message
– Bug: Fixed HTML Source scrapping issue
10/21/2019 (v1. 1. 3) + Bug Fixes
07/06/2019 (v1. 1) + UI Changes
+ Fixed: Scheduled Posts
+ Added: Ajax data parsing
+ Fixed: Parsing XML Feeds
+ Added: Non-english language support
+ Fixed: Local timezone capability
03/20/2019 (v1. 1) + UI revamped, many changes across the plugin and visual editor
+ Introducing Simplified view in “Scheduled Tasks” tab
+ Clone task feature introduced.
02/10/2019 (v1. 42) + ignore_params database issue has been fixed.
02/07/2019 (v1. 41)
+ product attributes issue has been fixed.
+ Split delimiter bug fixed for gallery images.
01/12/2019 (v1. 40)
+ DeepL Translation integration.
+ Disable URL parameters for URL uniqueness option.
+ Set featured image with post parent ID.
+ Total run limit functionality has been added.
+ Set author option has been added.
+ Set post slug option has been added.
12/06/2018 (v1. 39)
+ Multiple p element issue has been fixed for post_content.
+ Page increase stuck issues has been fixed.
11/27/2018 (v1. 38)
+ (Bugfix) Used images won’t be downloaded again.
+ Next page jump and auto-reset functions has been improved.
+ cURL gzip enabled.
+ WooCommerce product_cat and product_tag issues has been fixed.
11/13/2018 (v1. 37)
+ Bulk list URL problems has been fixed.
+ URL path issues has been fixed.
+ Next page trigger functions has been updated.
+ Gallery selection issues has been fixed.
+ Visual selector has been improved.
+ Default value for product attributes has been updated.
+ Next page selector has been improved.
+ Article content selector has been improved.
+ cURL feature has been added.
+ Proxy issues has been fixed.
+ Visual editor zoom issue has been fixed for low resolutions.
+ product_tag creation issue has been fixed.
+ Same variable name issue has been fixed for visual editor.
11/02/2018 (v1. 36)
+ URL Path issues has been fixed.
+ Bulk URL list option added.
+ Yandex Translation API has been added.
+ tbody issue has been fixed for Chrome.
+ Post formats has been added on project settings.
+ Ajax request error has been updated.
+ Index selection issue has been fixed for complex tables.
+ New track option added: no change on post status.
+ Notices removed from settings area.
10/22/2018 (v1. 35)
+ Pagination issues has been fixed.
+ Image download and is required issues has been fixed.
10/19/2018 (v1. 34)
+ Memory issue has been fixed.
10/19/2018 (v1. 33)
+ Aliexpress image issue has been fixed.
+ Error display added for timeout and memory issues.
+ Remove element for removing popups, preloaders.
+ “Required” field added. It pass if field is empty.
+ Special Conditions section added.
+ Task limit issue has been fixed.
10/16/2018 (v1. 32)
+ Next page function on preview results.
+ Auto-detection for scraping feature has been added.
+ Using variables as shortcodes on WordPress is possible now!
+ Visual editor Scraping base URL problem fixed.
+ Pause task issue has been fixed. (test edilecek)
+ Proxy URL issue on Visual tool, fixed.
+ Excluding posts by attributes is possible now.
+ Detailed logs for issues, problems.
+ Connection test functionality added.
+ Extract ID, HTML or URL from image attachment.
+ Limit task trigger function.
+ Next page URL issues has been fixed for location redirections.
10/12/2018 (v1. 31)
+ For is multiple field, JSON database format added.
+ Split content by words option added.
+ File download as attachment added. (pdf)
+ SKU Uniqueness method added.
+ _stock, _manage_stock, _backorders attributes has added.
+ Track changes function has been updated with uniquness method.
+ Post title can be used on filenames.
10/10/2018 (v1. 30)
+ Post excerpt field added, it’s also short product description.
+ Post delete issues has been fixed.
+ Post status change functionality added.
10/09/2018 (v1. 29)
+ Proxy issues has been fixed on visual editor
+ Visual editor’s local copy is on plugin with latest version.
+ Long URL issues has been fixed.
+ Post uniqueness issue has been fixed for post updates.
+ Delete post with source URL changes feature added.
+ Task process delay added.
+ Stop task feature replaced with pause task. It continues from last index.
+ decode feature added.
+ Multiple custom field feature added.
+ Media library attachment problem fixed for image objects.
+ Some UI problems has been fixed.
09/28/2018 (v1. 28)
+ Filename template feature added for all images
+ Strip and download images option added for content images
09/28/2018 (v1. 27)
+ Database migration issue has been fixed
09/28/2018 (v1. 26)
+ Gallery shortcode options added
+ Post URL and title uniqueness mechanism rewritten
+ Category creation issues has been fixed
+ Each post URL shortcode added
09/26/2018 (v1. 25)
+ Base URL issues for some web sites has been fixed (Aliexpress)
+ Clean Advertising codes improved
09/25/2018 (v1. 24)
+ Custom field issues has been fixed.
+ Documentation updated
+ Gallery images and featured images shouldn’t be duplicated
09/24/2018 (v1. 23)
+ Multiple category selection feature added
+ Filename transform section enabled
+ Base URL issues has been fixed for proxy
+ Custom taxonomy featured added
+ Gallery has been fixed for woocommerce products
+ Split function added for tags (With delimiter)
+ Google Translate API errors display
+ Serial selector for category, tags and images
+ Variable renderer minor fixes
+ Create new category with data field
+ Visual Editor improvements, UI bugs has been fixed
09/20/2018 (v1. 22)
+ Strip only some tags (for advertising codes)
+ Attributes UX fix
+ Base URL issues has been fixed
+ Post dates added for custom field
09/20/2018 (v1. 21)
+ Multiple attributes for Woocommerce products
+ UTF8 problems fixed for Non-English languages
09/19/2018 (v1. 20)
+ Preg match issues fixed
+ Xpath selector improvements
+ Auto detection feature for serial links
09/17/2018 (v1. 19)
+ Next page selection
+ Design issues and improvements
+ Result preview
+ Performance updates
+ Custom field issue fixed
+ Numeric functions problem fixed
09/15/2018 (v1. 18)
+ 0 byte image issue fixed
+ Image gallery issues fixed
+ Xpath selector fixed and improved
+ Regex parser method changed with /g flag
+ Find and replace method improved, it supports multiple find&replace
+ Excluding URLs is possible with this version
+ {{source_url}} option added
+ PHP errors fixed
09/07/2018 (v1. 16)
+ RSS feed or XML parsing method added.
+ Woocommerce product category issue fixed.
09/05/2018 (v1. 13)
+ Image Gallery Parse
+ UX Issue: single post checkbox lock
+ Strip only tags
+ A. I. Spinner Integration Error
+ Style Parser
09/05/2018 (v1. 12)
+ Trigger Issues Fixed
+ More Templates Added
+ Featured Image issues fixed
+ Cross-origin issues fixed
+ Xpath selector improved
08/17/2018 (v1. 0)
+ Cron issues fixed
+ Image download feature added
+ JSON Parsing Method Added
+ A. Spinner Service Enabled
+ Translation Service Enabled
+ First Release
Notes
On this section, we’ll be helping you about your concerns before you purchase.
Your hosting or server should allow incoming and outgoing network transfers for scraping.
Our support team can reach task’s content from service logs if you provide us any. This method makes easier support process and with this way our support team assist you quickly on any case.
We’ll be updating plugin for any bug or features that requested by users. Every update has been tested by our support team and softwares.
Plugin is licensed per domain. You should buy licence for each domain. You could move your licence to another domain.
Scraper doesn’t support ajax websites because of technical limitations.
Our plugin uses Google Translate official API unlike other plugins. So please make sure you have API key on that service before you purchase for translation services. Please also note that Google Translate’s Official API is a paid service.
Scraper’s plugin has functions that scrapes elements from given URLs. Our visual editor is a cloud based editor that hosted by us on our web service. Plugin doesn’t contains visual editor’s files in it. It only gets JSON data that built on visual editor.
Since Amazon blocked our servers, the plugin is unable to scrape any product from Amazon. That might be same for AliExpress and other big websites in future. Please make sure that plugin works properly before purchase. You could simply contact with our support team or comment and ask about support status. We’ll gladly test and inform you about changes.
You could request refund on any case that you are not satisfied on plugin.