How to Use Socks5 Proxy in Curl – Yuanle’s blog
Created On: 2016-12-22
Curl has the best proxy support among many HTTP clients and download
tools. This is how you use a socks5 proxy and also resolve hostname in the URL
using the socks5 proxy. For some use case, resolving hostname via the proxy is
essential.
Suppose you have a socks5 proxy running on localhost:8001.
In curl >= 7. 21. 7, you can use
curl -x socks5hlocalhost:8001
In curl >= 7. 18. 0, you can use
curl –socks5-hostname localhost:8001
Many tools use libcurl internally or use curl command in their installer
script. If it’s difficult to modify the command line itself, you can set proxy
using environment variables.
env ALL_PROXY=socks5hlocalhost:8001 PROGRAM [OPTION]…
If you want to overwrite system proxy settings, you may also need to set two
more variables:
env _proxy=socks5hlocalhost:8001 HTTPS_PROXY=socks5hlocalhost:8001 ALL_PROXY=socks5hlocalhost:8001 PROGRAM [OPTION]…
Note that _proxy is lower case, the other two is upper case.
curl ootw: –socks5 – Daniel Stenberg
(Previous option of the week posts. )
–socks5 was added to curl back in 7. 18. 0. It takes an argument and that argument is the host name (and port number) of your SOCKS5 proxy server. There is no short option version.
Proxy
A proxy, often called a forward proxy in the context of clients, is a server that the client needs to connect to in order to reach its destination. A middle man/server that we use to get us what we want. There are many kinds of proxies. SOCKS is one of the proxy protocols curl supports.
SOCKS
SOCKS is a really old proxy protocol. SOCKS4 is the predecessor protocol version to SOCKS5. curl supports both and the newer version of these two, SOCKS5, is documented in RFC 1928 dated 1996! And yes: they are typically written exactly like this, without any space between the word SOCKS and the version number 4 or 5.
One of the more known services that still use SOCKS is Tor. When you want to reach services on Tor, or the web through Tor, you run the client on your machine or local network and you connect to that over SOCKS5.
Which one resolves the host name
One peculiarity with SOCKS is that it can do the name resolving of the target server either in the client or have it done by the proxy. Both alternatives exists for both SOCKS versions. For SOCKS4, a SOCKS4a version was created that has the proxy resolve the host name and for SOCKS5, which is really the topic of today, the protocol has an option that lets the client pass on the IP address or the host name of the target server.
The –socks5 option makes curl itself resolve the name. You’d instead use –socks5-hostname if you want the proxy to resolve it.
–proxy
The –socks5 option is basically considered obsolete since curl 7. 21. 7. This is because starting in that release, you can now specify the proxy protocol directly in the string that you specify the proxy host name and port number with already. The server you specify with –proxy. If you use a socks5 scheme, curl will go with SOCKS5 with local name resolve but if you instead use socks5h it will pick SOCKS5 with proxy-resolved host name.
SOCKS authentication
A SOCKS5 proxy can also be setup to require authentication, so you might also have to specify name and password in the –proxy string, or set separately with –proxy-user. Or with GSSAPI, so curl also supports –socks5-gssapi and friends.
Examples
Fetch HTTPS from over the SOCKS5 proxy at port 1080. Remember that –socks5 implies that curl resolves the host name itself and passes the address to to use to the proxy.
curl –socks5
Or download FTP over the SOCKS5 proxy at socks5. example port 9999:
curl –socks5 socks5. example:9999
Useful trick!
A very useful trick that involves a SOCKS proxy is the ability OpenSSH has to create a SOCKS tunnel for us. If you sit at your friends house, you can open a SOCKS proxy to your home machine and access the network via that. Like this. First invoke ssh, login to your home machine and ask it to setup a SOCKS proxy:
ssh -D 8080
Then tell curl (or your browser, or both) to use this new SOCKS proxy when you want to access the Internet:
curl –socks5 localhost:8080 /
This will effectively hide all your Internet traffic from your friends snooping and instead pass it all through your encrypted ssh tunnel.
Related options
As already mentioned above, –proxy is typically the preferred option these days to set the proxy. But –socks5-hostname is there too and the related –socks4 and –sock4a.
tech, open source and networking
How to use cURL with proxy? – Blog | Oxylabs
This step-by-step guide will explain how to use cURL or simply, curl, with proxy servers. It covers all the aspects, beginning from installation to explaining various options to set the proxy.
We did not target any specific proxy service. Therefore this tutorial should work with all proxy servers. All you need to know is the server details and credentials.
This is a fairly technical tutorial and expects readers to have a basic understanding of what a proxy is. It would be especially interesting and useful for those starting with web scraping.
Navigation:
What is cURL?
Installation
What you need to connect to a proxy
Using cURL with HTTP/HTTPS proxy
Command line argument to set proxy in cURL
Using environment variables
Configure cURL to always use proxy
Ignore or override proxy for one request
Bonus tip – turning proxies off and on quickly
cURL socks proxy
Summary
cURL is a command line tool for sending and receiving data using the url. Let’s look at the simplest example of using curl. Open your terminal or command prompt and type in this command and press Enter:
curl This will get the HTML of the page and print it on the console.
curl -I
This will print the document information.
HTTP/1. 1 200 OK
Content-Type: text/html; charset=ISO-8859-1
The question “what is cURL? ” is also answered in one of our previous articles. We recommend reading it if you want to learn how it became such a universal asset.
cURL is provided with many Linux distributions and with MacOS. Now it is provided with Windows 10 as well.
If your Linux distribution is not provided with it, you can install it by running the install command. For example, on Ubuntu, open Terminal and run this command:
sudo apt install curl
If you are running an older version of Windows, or if you want to install an alternate version, you can download curl from the official download page.
Irrespective of which proxy service you use, you will need the following information to use a:
proxy server addressportprotocolusername (if authentication is required)password (if authentication is required)
In this tutorial, we are going to assume that the proxy server is 127. 0. 1, the port is 1234, the user name is user, and the password is pwd. We will look into multiple examples covering various protocols.
NOTE. If you are on a network that uses NTLM authentication, you can use the switch –proxy-ntlm while running curl. Similarly, –proxy-digest can be used for digest authentication. You can look at all the available options by running curl –help. This tutorial will have examples for the scenario when a username and password has to be specified.
The next section will cover the first curl proxy scenario, which happens to be the most common one – HTTP and HTTPS proxy with curl.
If you recall, we looked at using curl without proxy like this:
curl This particular website is especially useful for testing out proxies as the output of this page is the origin IP address. If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address.
There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.
NOTE. All the command line options, or switches, are case sensitive. For example, -f instructs curl to fail silently, while -F denotes a form to be submitted.
Open terminal and type the following command, and press Enter:
curl –help
The output is going to be a huge list of options. One of them is going to look like this:
-x, –proxy [protocol]host[:port]
Note that x is small, and it is case-sensitive. The proxy details can be supplied using -x or –proxy switch. Both mean the same thing. Bot of the curl with proxy commands are same:
curl -x “user:pwd@127. 1:1234″ ”
or
curl –proxy “user:pwd@127. 1:1234″ ”
NOTE. If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.
curl –proxy “user:pwd@127. 1:1234″ ” -k
You may have noticed that both the proxy url and target url are surrounded in double quotes. This is a recommended practice to handle special characters in the url.
Another interesting thing to note here is that the default proxy protocol is. Thus, following two commands will do exactly the same:
Another way to use proxy with curl is to set the environment variables _proxy and _proxy.
Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc file.
If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.
_proxy – the proxy will be used to access addresses that use protocol
Simply set the variables _proxy to proxy address and _proxy to set proxy address. Open terminal and run these two commands.
export _proxy=”user:pwd@127. 1:1234″
After running these two commands, run curl normally.
curl ”
If you see SSL Certificate errors, add -k to ignore these errors.
Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:
unset _proxy
See the next section to set default proxy only for curl and not system wide.
If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.
For Linux and MacOS, open terminal and navigate to your home directory. If there is already a file, open it. If there is none, create a new file. Here are the set of commands that can be run:
cd ~
nano
In this file, add this line:
proxy=”user:pwd@127. 1:1234″
Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from file.
On Windows, the file is named _curlrc. This file can be placed in the%APPDATA% directory.
To find the exact path of%APPDATA%, open command prompt and run the following command:
echo%APPDATA%
This directory will be something like C:Users
This works exactly the same way in Linux, MacOS, and Windows.
If the proxy is set globally, or by modifying the file, this can still be overridden to set another proxy or even bypass it.
To override proxy for one request, set the new proxy using -x or –proxy switch as usual:
curl –proxy “user:pwd@1. 1:8090″ ”
If you want to bypass proxy altogether for a request, you can pass –noproxy followed by “*”. This instructs curl to not use proxy for all URLs.
curl –noproxy “*” ”
If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.
This tip is dedicated only for advanced users. If you do not know what a file is, you may skip this section.
You can create an alias in your file to set proxies and unset proxies. For example, open file using any editor and add these lines:
alias proxyon=”export _proxy=’ user:pwd@127. 1:1234′;export _proxy=’ user:pwd@127. 1:1234′”
alias proxyoff=”unset _proxy;unset _proxy”
After adding these lines, save the and update the shell to read this To do this, run this this command in the terminal:. ~/
Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:
proxyon
proxyoff
If the proxy server is using socks protocol, the syntax remains the same:
curl -x “socks5user:pwd@127. 1:1234″ ”
Similarly, socks4, socks4a, socks5 or socks5h can be used depending on the socks version.
Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.
curl –socks5 “127. 1:1234″ ” –proxy-user user:pwd
Again, –socks4, –socks4a or –socks5 can be used, depending on the version.
cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.
You can learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.
Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you’ll probably find her just chillin’ while listening to her favorite music or playing board games with friends.
All information on Oxylabs Blog is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.
Frequently Asked Questions about curl socks proxy
What proxy does curl use?
curl supports several different types of proxies. The default proxy type is HTTP so if you specify a proxy host name (or IP address) without a scheme part (the part that is often written as “http://”) curl goes with assuming it’s an HTTP proxy.
Is socks and proxy the same?
Unlike HTTP proxies, which can only interpret and work with HTTP and HTTPS webpages, SOCKS5 proxies can work with any traffic. HTTP proxies are high-level proxies usually designed for a specific protocol. … SOCKS proxies are low-level proxies that can handle any program or protocol and any traffic without limitations.
How do I use a proxy to curl?
On Curl for example you can set the proxy using the –proxy flag:curl http://example.com –proxy 127.0.0.1:8080. Sh.proxy = 127.0.0.1:8080.http_proxy = http://127.0.0.1:8080.Apr 17, 2018