Very Simple C++ Web Crawler/Spider? – Stack Overflow
All right, I’ll try to point you in the right direction. Conceptually, a webcrawler is pretty simple. It revolves around a FIFO queue data structure which stores pending URLs. C++ has a built-in queue structure in the standard libary, std::queue, which you can use to store URLs as strings.
The basic algorithm is pretty straightforward:
Begin with a base URL that you
select, and place it on the top of
your queue
Pop the URL at the top of the queue
and download it
Parse the downloaded HTML file and extract all links
Insert each extracted link into the queue
Goto step 2, or stop once you reach some specified limit
Now, I said that a webcrawler is conceptually simple, but implementing it is not so simple. As you can see from the above algorithm, you’ll need: an HTTP networking library to allow you to download URLs, and a good HTML parser that will let you extract links. You mentioned you could use wget to download pages. That simplifies things somewhat, but you still need to actually parse the downloaded HTML docs. Parsing HTML correctly is a non-trivial task. A simple string search for
C++ A Simple Web Crawler – Chilkat Example Code
This demonstrates a very simple web crawler using the Chilkat Spider component.
#include
#include
void ChilkatSample(void)
{
CkSpider spider;
CkStringArray seenDomains;
CkStringArray seedUrls;
seenDomains. put_Unique(true);
seedUrls. put_Unique(true);
// You will need to change the start URL to something else…
(“);
// Set outbound URL exclude patterns
// URLs matching any of these patterns will not be added to the
// collection of outbound links.
dAvoidOutboundLinkPattern(“*? id=*”);
dAvoidOutboundLinkPattern(“*. mypages. *”);
dAvoidOutboundLinkPattern(“*. personal. comcast. *”);
dAvoidOutboundLinkPattern(“**”);
dAvoidOutboundLinkPattern(“*~*”);
// Use a cache so we don’t have to re-fetch URLs previously fetched.
spider. put_CacheDir(“c:/spiderCache/”);
spider. put_FetchFromCache(true);
spider. put_UpdateCache(true);
while (t_Count() > 0) {
const char *url = ();
itialize(url);
// Spider 5 URLs of this domain.
// but first, save the base domain in seenDomains
const char *domain = tUrlDomain(url);
(tBaseDomain(domain));
int i;
bool success;
for (i = 0; i <= 4; i++) {
success = awlNext();
if (success == true) {
// Display the URL we just crawled.
std::cout << stUrl() << "rn";
// If the last URL was retrieved from cache,
// we won't wait. Otherwise we'll wait 1 second
// before fetching the next URL.
if (t_LastFromCache()! = true) {
eepMs(1000);}}
else {
// cause the loop to exit..
i = 999;}}
// Add the outbound links to seedUrls, except
// for the domains we've already seen.
for (i = 0; i <= t_NumOutboundLinks() - 1; i++) {
url = tOutboundLink(i);
const char *baseDomain = tBaseDomain(domain);
if (ntains(baseDomain) == false) {
// Don't let our list of seedUrls grow too large.
if (t_Count() < 1000) {
(url);}}}}}
©
2000-2021 Chilkat Software, Inc. All Rights Reserved.
Simplest Possible Web Crawler with C++ – gist GitHub
//============================================================================
// Name:
// Author: Berlin Brown (berlin dot brown at)
// Version:
// Copyright: Copyright Berlin Brown 2012-2013
// License: BSD
// Description: This is the simplest possible web crawler in C++
// Uses boost_regex and boost_algorithm
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
using namespace std;
using namespace boost;
const int DELAY = 12;
const int MAXRECV = 140 * 1024;
const std::string WRITE_DIR_PATH = “/home/bbrown/public/”;
class WebPage {
public:
std::string hostname;
std::string page;
WebPage() {
hostname = “”;
page = “”;}
std::string parseHttp(const std::string str) {
const boost::regex re(“(? i)(. *)/? (. *)”);
boost::smatch what;
if (boost::regex_match(str, what, re)) {
std::string hst = what[1];
boost::algorithm::to_lower(hst);
return hst;}
return “”;} // End of method //
void parseHref(const std::string orig_host, const std::string str) {
const boost::regex re(“(? i)(. *)/(. *)”);
// We found a full URL, parse out the ‘hostname’
// Then parse out the page
hostname = what[1];
boost::algorithm::to_lower(hostname);
page = what[2];} else {
// We could not find the ‘page’ but we can build the hostname
hostname = orig_host;
page = “”;} // End of the if – else //} // End of method //
void parse(const std::string orig_host, const std::string hrf) {
const std::string hst = parseHttp(hrf);
if (! ()) {
// If we have a HTTP prefix
// We could end up with a ‘hostname’ and page
parseHref(hst, hrf);} else {
page = hrf;}
// hostname and page are constructed,
// perform post analysis
if (() == 0) {
page = “/”;} // End of the if //} // End of the method}; // End of the class
std::string string_format(const std::string &fmt,… ) {
int size = 255;
std::string str;
va_list ap;
while (1) {
(size);
va_start(ap, fmt);
int n = vsnprintf((char *) str. c_str(), size, fmt. c_str(), ap);
va_end(ap);
if (n > -1 && n < size) {
(n);
return str;}
if (n > -1)
size = n + 1;
else
size *= 2;} // End of the while //
return str;} // End of the function //
std::string request(std::string host, std::string path) {
std::string request = “GET “;
(path);
(” HTTP/1. 1rn”);
(“Host: “);
(host);
(“rn”);
(“Accept: text/html, application/xhtml+xml, application/xml;q=0. 9, */*;q=0. 81rn”);
(“User-Agent: Mozilla/5. 0 (compatible; octanebot/1. 0;)rn”);
(“Connection: closern”);
return request;} // End of the function //
std::string clean_href(const std::string host, const std::string path) {
// Clean the href to save to file //
std::string full_url = host;
(“/”);
const boost::regex rmv_all(“[^a-zA-Z0-9]”);
const std::string s2 = boost::regex_replace(full_url, rmv_all, “_”);
cout << s2 << endl;
return s2;}
int connect(const std::string host, const std::string path) {
const int port = 80;
// Setup the msock
int m_sock;
sockaddr_in m_addr;
memset(&m_addr, 0, sizeof(m_addr));
m_sock = socket(AF_INET, SOCK_STREAM, 0);
int on = 1;
if (setsockopt(m_sock, SOL_SOCKET, SO_REUSEADDR, (const char*) &on, sizeof(on)) == -1) {
return false;}
// Connect //
n_family = AF_INET;
n_port = htons(port);
int status = inet_pton(AF_INET, host. c_str(), &n_addr);
if (errno == EAFNOSUPPORT) {
status =::connect(m_sock, (sockaddr *) &m_addr, sizeof(m_addr));
// HTTP/1. 1 defines the "close" connection option for
// the sender to signal that the connection will be closed
// after completion of the response.
std::string req = request(host, path);
// End of building the request //
status =::send(m_sock, req. c_str(), (), MSG_NOSIGNAL);
char buf[MAXRECV];
cout << "Request: " << req << endl;
cout << "=========================" << endl;
std::string recv = "";
while (status! = 0) {
memset(buf, 0, MAXRECV);
status =::recv(m_sock, buf, MAXRECV, 0);
(buf);} // End of the while //
cout << "Response:" << recv << endl;
cout << "---------------------------" << endl;
// Attempt to write to file //
const std::string html_file_write = string_format("%s/%s", WRITE_DIR_PATH. c_str(), clean_href(host, path). c_str());
cout << "Writing to file: " << html_file_write << endl;
ofstream outfile(html_file_write. c_str());
outfile << recv << endl;
();
// Parse the data //
try {
const boost::regex rmv_all("[\r|\n]");
const std::string s2 = boost::regex_replace(recv, rmv_all, "");
const std::string s = s2;
// Use this regex expression, allow for mixed-case
// Search for the anchor tag but not the '>‘
// Where (. +? ) match anything
//const boost::regex re(“
const boost::regex re(“