Use a Proxy (TLDR: Use -x argument) – Curl Cookbook
These curl recipes show you how to proxy your curl requests through a socks4, socks5, or a regular (or) proxy. To set a proxy, use the -x protocoluser:password@host:port command line argument.
Use a Socks5 Proxy
Use a Socks4 Proxy
Use an HTTP Proxy
Don’t Use a Proxy for
curl -x socks5 In this recipe, curl uses the -x argument to set the proxy protocol to socks5, proxy username to james, proxy password to cats, proxy hostname to, and proxy port to 8080. After connecting to the proxy, curl then makes a GET request to Use a Socks4 Proxy
curl -x socks4 This recipe is a carbon copy of the previous recipe. The only change is that the protocol has been set to socks4.
curl -x In this recipe, the protocol part of the proxy address is not specified. In this case, the HTTP proxy is assumed. Just like in the previous two recipes, the username and password are set to james:cats and the proxy host and port are set to
curl –no-proxy -x In this recipe, the –no-proxy command line argument is used. This argument prevents curl from proxying requests to The next argument is -x that sets the HTTPS proxy for curl to (without username or password). After this, curl makes three requests to,, and. The first two requests are proxied through but the last request is not proxied because matches the proxy ignore list.
Created by Browserling
These curl recipes were written down by me and my team at Browserling. We use recipes like this every day to get things done and improve our product. Browserling itself is an online cross-browser testing service powered by alien technology. Check it out!
Secret message: If you love my curl recipe, then I love you, too! Use coupon code CURLLING to get a discount at my company.
How to use cURL with proxy? – Blog | Oxylabs
This step-by-step guide will explain how to use cURL or simply, curl, with proxy servers. It covers all the aspects, beginning from installation to explaining various options to set the proxy.
We did not target any specific proxy service. Therefore this tutorial should work with all proxy servers. All you need to know is the server details and credentials.
This is a fairly technical tutorial and expects readers to have a basic understanding of what a proxy is. It would be especially interesting and useful for those starting with web scraping.
Navigation:
What is cURL?
Installation
What you need to connect to a proxy
Using cURL with HTTP/HTTPS proxy
Command line argument to set proxy in cURL
Using environment variables
Configure cURL to always use proxy
Ignore or override proxy for one request
Bonus tip – turning proxies off and on quickly
cURL socks proxy
Summary
cURL is a command line tool for sending and receiving data using the url. Let’s look at the simplest example of using curl. Open your terminal or command prompt and type in this command and press Enter:
curl This will get the HTML of the page and print it on the console.
curl -I
This will print the document information.
HTTP/1. 1 200 OK
Content-Type: text/html; charset=ISO-8859-1
The question “what is cURL? ” is also answered in one of our previous articles. We recommend reading it if you want to learn how it became such a universal asset.
cURL is provided with many Linux distributions and with MacOS. Now it is provided with Windows 10 as well.
If your Linux distribution is not provided with it, you can install it by running the install command. For example, on Ubuntu, open Terminal and run this command:
sudo apt install curl
If you are running an older version of Windows, or if you want to install an alternate version, you can download curl from the official download page.
Irrespective of which proxy service you use, you will need the following information to use a:
proxy server addressportprotocolusername (if authentication is required)password (if authentication is required)
In this tutorial, we are going to assume that the proxy server is 127. 0. 1, the port is 1234, the user name is user, and the password is pwd. We will look into multiple examples covering various protocols.
NOTE. If you are on a network that uses NTLM authentication, you can use the switch –proxy-ntlm while running curl. Similarly, –proxy-digest can be used for digest authentication. You can look at all the available options by running curl –help. This tutorial will have examples for the scenario when a username and password has to be specified.
The next section will cover the first curl proxy scenario, which happens to be the most common one – HTTP and HTTPS proxy with curl.
If you recall, we looked at using curl without proxy like this:
curl This particular website is especially useful for testing out proxies as the output of this page is the origin IP address. If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address.
There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.
NOTE. All the command line options, or switches, are case sensitive. For example, -f instructs curl to fail silently, while -F denotes a form to be submitted.
Open terminal and type the following command, and press Enter:
curl –help
The output is going to be a huge list of options. One of them is going to look like this:
-x, –proxy [protocol]host[:port]
Note that x is small, and it is case-sensitive. The proxy details can be supplied using -x or –proxy switch. Both mean the same thing. Bot of the curl with proxy commands are same:
curl -x “user:pwd@127. 1:1234″ ”
or
curl –proxy “user:pwd@127. 1:1234″ ”
NOTE. If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.
curl –proxy “user:pwd@127. 1:1234″ ” -k
You may have noticed that both the proxy url and target url are surrounded in double quotes. This is a recommended practice to handle special characters in the url.
Another interesting thing to note here is that the default proxy protocol is. Thus, following two commands will do exactly the same:
Another way to use proxy with curl is to set the environment variables _proxy and _proxy.
Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc file.
If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.
_proxy – the proxy will be used to access addresses that use protocol
Simply set the variables _proxy to proxy address and _proxy to set proxy address. Open terminal and run these two commands.
export _proxy=”user:pwd@127. 1:1234″
After running these two commands, run curl normally.
curl ”
If you see SSL Certificate errors, add -k to ignore these errors.
Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:
unset _proxy
See the next section to set default proxy only for curl and not system wide.
If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.
For Linux and MacOS, open terminal and navigate to your home directory. If there is already a file, open it. If there is none, create a new file. Here are the set of commands that can be run:
cd ~
nano
In this file, add this line:
proxy=”user:pwd@127. 1:1234″
Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from file.
On Windows, the file is named _curlrc. This file can be placed in the%APPDATA% directory.
To find the exact path of%APPDATA%, open command prompt and run the following command:
echo%APPDATA%
This directory will be something like C:Users
This works exactly the same way in Linux, MacOS, and Windows.
If the proxy is set globally, or by modifying the file, this can still be overridden to set another proxy or even bypass it.
To override proxy for one request, set the new proxy using -x or –proxy switch as usual:
curl –proxy “user:pwd@1. 1:8090″ ”
If you want to bypass proxy altogether for a request, you can pass –noproxy followed by “*”. This instructs curl to not use proxy for all URLs.
curl –noproxy “*” ”
If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.
This tip is dedicated only for advanced users. If you do not know what a file is, you may skip this section.
You can create an alias in your file to set proxies and unset proxies. For example, open file using any editor and add these lines:
alias proxyon=”export _proxy=’ user:pwd@127. 1:1234′;export _proxy=’ user:pwd@127. 1:1234′”
alias proxyoff=”unset _proxy;unset _proxy”
After adding these lines, save the and update the shell to read this To do this, run this this command in the terminal:. ~/
Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:
proxyon
proxyoff
If the proxy server is using socks protocol, the syntax remains the same:
curl -x “socks5user:pwd@127. 1:1234″ ”
Similarly, socks4, socks4a, socks5 or socks5h can be used depending on the socks version.
Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.
curl –socks5 “127. 1:1234″ ” –proxy-user user:pwd
Again, –socks4, –socks4a or –socks5 can be used, depending on the version.
cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.
You can learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.
Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you’ll probably find her just chillin’ while listening to her favorite music or playing board games with friends.
All information on Oxylabs Blog is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.
Proxies – Everything curl
A proxy is a machine or software that does something on behalf of you, the can also see it as a middle man that sits between you and the server you want to work with, a middle man that you connect to instead of the actual remote server. You ask the proxy to perform your desired operation for you and then it will run off and do that and then return the data to are several different types of proxies and we shall list and discuss them further down in this networks are setup to require a proxy in order for you to reach the Internet or perhaps that special network you are interested in. The use of proxies are introduced on your network by the people and management that run your network for policy or technical the networking space there are a few methods for the automatic detection of proxies and how to connect to them, but none of those methods are truly universal and curl supports none of them. Furthermore, when you communicate to the outside world through a proxy that often means that you have to put a lot of trust on the proxy as it will be able to see and modify all the non-secure network traffic you send or get through it. That trust is not easy to assume you check your browser’s network settings, sometimes under an advanced settings tab, you can learn what proxy or proxies your browser is configured to use. Chances are big that you should use the same one or ones when you use an example, you can find proxy settings for Firefox browser in Preferences => General => Network Settings as shown below:proxy settings for FirefoxSome network environments provides several different proxies that should be used in different situations, and a customizable way to handle that is supported by the browsers. This is called “proxy auto-config”, or PAC. A PAC file contains a JavaScript function that decides which proxy a given network connection (URL) should use, and even if it should not use a proxy at all. Browsers most typically read the PAC file off a URL on the local curl has no JavaScript capabilities, curl does not support PAC files. If your browser and network use PAC files, the easiest route forward is usually to read the PAC file manually and figure out the proxy you need to specify to run curl are not proxies but they’re blocking the way between you and the server you want to access. A “captive portal” is one of these systems that are popular to use in hotels, airports and for other sorts of network access to a larger audience. The portal will “capture” all network traffic and redirect you to a login web page until you have either clicked OK and verified that you have read their conditions or perhaps even made sure that you have paid plenty of money for the right to use the ‘s traffic will of course also captured by such portals and often the best way is to use a browser to accept the conditions and “get rid of” the portal since from then on they often allow all other traffic originating from that same machine (MAC address) for a period of often you can use curl too to submit that “ok” affirmation, if you just figure out how to submit the form and what fields to include in it. If this is something you end up doing many times, it may be worth supports several different types of default proxy type is HTTP so if you specify a proxy host name (or IP address) without a scheme part (the part that is often written as “”) curl goes with assuming it’s an HTTP also allows a number of different options to set the proxy type instead of using the scheme prefix. See the SOCKS section HTTP proxy is a proxy that the client speaks HTTP with to get the transfer done. curl will, by default, assume that a host you point out with -x or –proxy is an HTTP proxy, and unless you also specify a port number it will default to port 1080 (and the reason for that particular port number is purely historical) you want to request the web page using a proxy on 192. 168. 0. 1 port 8080, a command line could look like:curl -x 192. 1:8080 that the proxy receives your request, forwards it to the real server, then reads the response from the server and then hands that back to the you enable verbose mode with -v when talking to a proxy, you will see that curl connects to the proxy instead of the remote server, and you will see that it uses a slightly different request was designed to allow and provide secure and safe end-to-end privacy from the client to the server (and back). In order to provide that when speaking to an HTTP proxy, the HTTP protocol has a special request that curl uses to setup a tunnel through the proxy that it then can encrypt and verify. This HTTP method is known as the proxy tunnels encrypted data through to the remote server after a CONNECT method sets it up, the proxy cannot see nor modify the traffic without breaking the encryption:curl -x means Man-In-The-Middle. MITM-proxies are usually deployed by companies in “enterprise environments” and elsewhere, where the owners of the network have a desire to investigate even TLS encrypted do this, they require users to install a custom “trust root” (Certificate Authority (CA) certificate) in the client, and then the proxy terminates all TLS traffic from the client, impersonates the remote server and acts like a proxy. The proxy then sends back a generated certificate signed by the custom CA. Such proxy setups usually transparently capture all traffic from clients to TCP port 443 on a remote machine. Running curl in such a network would also get its HTTPS traffic practice, of course, allows the middle man to decrypt and snoop on all TLS “HTTP proxy” means the proxy itself speaks HTTP. HTTP proxies are primarily used to proxy HTTP but it is also fairly common that they support other protocols as well. In particular, FTP is fairly commonly talking FTP “over” an HTTP proxy, it is usually done by more or less pretending the other protocol works like HTTP and asking the proxy to “get this URL” even if the URL is not using HTTP. This distinction is important because it means that when sent over an HTTP proxy like this, curl does not really speak FTP even though given an FTP URL; thus FTP-specific features will not work:curl -x you can do instead then, is to “tunnel through” the HTTP proxy! Most HTTP proxies allow clients to “tunnel through” it to a server on the other side. That’s exactly what’s done every time you use HTTPS through the HTTP tunnel through an HTTP proxy with curl using -p or you do HTTPS through a proxy you normally connect through to the default HTTPS remote TCP port number 443, so therefore you will find that most HTTP proxies white list and allow connections only to hosts on that port number and perhaps a few others. Most proxies will deny clients from connecting to just any random port (for reasons only the proxy administrators know), assuming that the HTTP proxy allows it, you can ask it to tunnel through to a remote server on any port number so you can do other protocols “normally” even when tunneling. You can do FTP tunneling like this:curl -p -x can tell curl to use HTTP/1. 0 in its CONNECT request issued to the HTTP proxy by using –proxy1. 0 [proxy] instead of is a protocol used for proxies and curl supports it. curl supports both SOCKS version 4 as well as version 5, and both versions come in two can select the specific SOCKS version to use by using the correct scheme part for the given proxy host with -x, or you can specify it with a separate option instead of CKS4 is for the version 4 and SOCKS4a is for the version 4 without resolving the host name locally:curl -x socks4 curl –socks4 SOCKS4a versions:curl -x socks4a curl –socks4a is for the version 5 and SOCKS5-hostname is for the version 5 without resolving the host name locally:curl -x socks5 curl –socks5 SOCKS5-hostname versions. This sends the host name to the server so there’s no name resolving done locally:curl -x socks5h curl –socks5-hostname proxies can require authentication, so curl then needs to provide the proper credentials to the proxy to be allowed to use it, and failing to do will only make the proxy return HTTP responses using code thentication for proxies is similar to “normal” HTTP authentication. It is separate from the server authentication to allow clients to independently use both normal host authentication as well as proxy curl, you set the user name and password for the proxy authentication with the -U user:password or –proxy-user user:password option:curl -U daniel:secr3t -x myproxy:80 example will default to using the Basic authentication scheme. Some proxies will require another authentication scheme (and the headers that are returned when you get a 407 response will tell you which) and then you can ask for a specific method with –proxy-digest, –proxy-negotiate, –proxy-ntlm. The above example command again, but asking for NTLM auth with the proxy:curl -U daniel:secr3t -x myproxy:80 –proxy-ntlmThere’s also the option that asks curl to figure out which method the proxy wants and supports and then go with that (with the possible expense of extra roundtrips) using –proxy-anyauth. Asking curl to use any method the proxy wants is then like this:curl -U daniel:secr3t -x myproxy:80 –proxy-anyauthAll the previously mentioned protocols to speak with the proxy are clear text protocols, HTTP and the SOCKS versions. Using these methods could allow someone to eavesdrop on your traffic the local network where you or the proxy solution for that is to use HTTPS to the proxy, which then establishes a secure and encrypted connection that is safe from easy a HTTPS proxy is specified, the default port will be checks for the existence of specially named environment variables before it runs to see if a proxy is requested to get specify the proxy by setting a variable named [scheme]_proxy to hold the proxy host name (the same way you would specify the host with -x). So if you want to tell curl to use a proxy when access a HTTP server, you set the ‘_proxy’ environment variable. Like this__proxy= -v the above example shows HTTP, you can, of course, also set ftp_proxy, _proxy, and so on. All these proxy environment variable names except _proxy can also be specified in uppercase, like set a single variable that controls all protocols, the ALL_PROXY exists. If a specific protocol variable one exists, such a one will take using environment variables to set a proxy, you could easily end up in a situation where one or a few host names should be excluded from going through the proxy. This is then done with the NO_PROXY variable. Set that to a comma- separated list of host names that should not use a proxy when being accessed. You can set NO_PROXY to be a single asterisk (‘*’) to match all an alternative to the NO_PROXY variable, there’s also a –noproxy command line option that serves the same purpose and works the same HTTP version of the proxy environment variables is treated differently than the others. It is only accepted in its lower case version because of the CGI protocol, which lets users run scripts in a server when invoked by an HTTP server. When a CGI script is invoked by a server, it automatically creates environment variables for the script based on the incoming headers in the request. Those environment variables are prefixed with uppercase HTTP_! An incoming request to a HTTP server using a request header like Proxy: yada will therefore create the environment variable HTTP_PROXY set to contain yada before the CGI script is started. If that CGI script runs cepting the upper case version of this environment variable has been the source for many security problems in lots of software through you want to add HTTP headers meant specifically for a proxy and not for the remote server, the –header option falls example, if you issue a HTTPS request through a HTTP proxy, it will be done by first issuing a CONNECT to the proxy that establishes a tunnel to the remote server and then it sends the request to that server. That first CONNECT is only issued to the proxy and you may want to make sure only that receives your special header, and send another set of custom headers to the remote a specific different User-Agent: only to the proxy:curl –proxy-header “User-Agent: magic/3000” -x proxy
Frequently Asked Questions about curl socks4
What does the curl command do?
cURL, which stands for client URL, is a command line tool that developers use to transfer data to and from a server. At the most fundamental, cURL lets you talk to a server by specifying the location (in the form of a URL) and the data you want to send.Feb 23, 2021
Does curl use Http_proxy?
An HTTP proxy is a proxy that the client speaks HTTP with to get the transfer done. curl will, by default, assume that a host you point out with -x or –proxy is an HTTP proxy, and unless you also specify a port number it will default to port 1080 (and the reason for that particular port number is purely historical).
Is curl using proxy?
Using cURL with HTTP/HTTPS proxy If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address. There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.Aug 9, 2021