How to use cURL with proxy? – Blog | Oxylabs
This step-by-step guide will explain how to use cURL or simply, curl, with proxy servers. It covers all the aspects, beginning from installation to explaining various options to set the proxy.
We did not target any specific proxy service. Therefore this tutorial should work with all proxy servers. All you need to know is the server details and credentials.
This is a fairly technical tutorial and expects readers to have a basic understanding of what a proxy is. It would be especially interesting and useful for those starting with web scraping.
Navigation:
What is cURL?
Installation
What you need to connect to a proxy
Using cURL with HTTP/HTTPS proxy
Command line argument to set proxy in cURL
Using environment variables
Configure cURL to always use proxy
Ignore or override proxy for one request
Bonus tip – turning proxies off and on quickly
cURL socks proxy
Summary
cURL is a command line tool for sending and receiving data using the url. Let’s look at the simplest example of using curl. Open your terminal or command prompt and type in this command and press Enter:
curl This will get the HTML of the page and print it on the console.
curl -I
This will print the document information.
HTTP/1. 1 200 OK
Content-Type: text/html; charset=ISO-8859-1
The question “what is cURL? ” is also answered in one of our previous articles. We recommend reading it if you want to learn how it became such a universal asset.
cURL is provided with many Linux distributions and with MacOS. Now it is provided with Windows 10 as well.
If your Linux distribution is not provided with it, you can install it by running the install command. For example, on Ubuntu, open Terminal and run this command:
sudo apt install curl
If you are running an older version of Windows, or if you want to install an alternate version, you can download curl from the official download page.
Irrespective of which proxy service you use, you will need the following information to use a:
proxy server addressportprotocolusername (if authentication is required)password (if authentication is required)
In this tutorial, we are going to assume that the proxy server is 127. 0. 1, the port is 1234, the user name is user, and the password is pwd. We will look into multiple examples covering various protocols.
NOTE. If you are on a network that uses NTLM authentication, you can use the switch –proxy-ntlm while running curl. Similarly, –proxy-digest can be used for digest authentication. You can look at all the available options by running curl –help. This tutorial will have examples for the scenario when a username and password has to be specified.
The next section will cover the first curl proxy scenario, which happens to be the most common one – HTTP and HTTPS proxy with curl.
If you recall, we looked at using curl without proxy like this:
curl This particular website is especially useful for testing out proxies as the output of this page is the origin IP address. If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address.
There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.
NOTE. All the command line options, or switches, are case sensitive. For example, -f instructs curl to fail silently, while -F denotes a form to be submitted.
Open terminal and type the following command, and press Enter:
curl –help
The output is going to be a huge list of options. One of them is going to look like this:
-x, –proxy [protocol]host[:port]
Note that x is small, and it is case-sensitive. The proxy details can be supplied using -x or –proxy switch. Both mean the same thing. Bot of the curl with proxy commands are same:
curl -x “user:pwd@127. 1:1234″ ”
or
curl –proxy “user:pwd@127. 1:1234″ ”
NOTE. If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.
curl –proxy “user:pwd@127. 1:1234″ ” -k
You may have noticed that both the proxy url and target url are surrounded in double quotes. This is a recommended practice to handle special characters in the url.
Another interesting thing to note here is that the default proxy protocol is. Thus, following two commands will do exactly the same:
Another way to use proxy with curl is to set the environment variables _proxy and _proxy.
Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc file.
If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.
_proxy – the proxy will be used to access addresses that use protocol
Simply set the variables _proxy to proxy address and _proxy to set proxy address. Open terminal and run these two commands.
export _proxy=”user:pwd@127. 1:1234″
After running these two commands, run curl normally.
curl ”
If you see SSL Certificate errors, add -k to ignore these errors.
Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:
unset _proxy
See the next section to set default proxy only for curl and not system wide.
If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.
For Linux and MacOS, open terminal and navigate to your home directory. If there is already a file, open it. If there is none, create a new file. Here are the set of commands that can be run:
cd ~
nano
In this file, add this line:
proxy=”user:pwd@127. 1:1234″
Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from file.
On Windows, the file is named _curlrc. This file can be placed in the%APPDATA% directory.
To find the exact path of%APPDATA%, open command prompt and run the following command:
echo%APPDATA%
This directory will be something like C:\Users\
This works exactly the same way in Linux, MacOS, and Windows.
If the proxy is set globally, or by modifying the file, this can still be overridden to set another proxy or even bypass it.
To override proxy for one request, set the new proxy using -x or –proxy switch as usual:
curl –proxy “user:pwd@1. 1:8090″ ”
If you want to bypass proxy altogether for a request, you can pass –noproxy followed by “*”. This instructs curl to not use proxy for all URLs.
curl –noproxy “*” ”
If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.
This tip is dedicated only for advanced users. If you do not know what a file is, you may skip this section.
You can create an alias in your file to set proxies and unset proxies. For example, open file using any editor and add these lines:
alias proxyon=”export _proxy=’ user:pwd@127. 1:1234′;export _proxy=’ user:pwd@127. 1:1234′”
alias proxyoff=”unset _proxy;unset _proxy”
After adding these lines, save the and update the shell to read this To do this, run this this command in the terminal:. ~/
Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:
proxyon
proxyoff
If the proxy server is using socks protocol, the syntax remains the same:
curl -x “socks5user:pwd@127. 1:1234″ ”
Similarly, socks4, socks4a, socks5 or socks5h can be used depending on the socks version.
Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.
curl –socks5 “127. 1:1234″ ” –proxy-user user:pwd
Again, –socks4, –socks4a or –socks5 can be used, depending on the version.
cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.
You can learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.
Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you’ll probably find her just chillin’ while listening to her favorite music or playing board games with friends.
All information on Oxylabs Blog is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.
performing HTTP requests with cURL (using PROXY) – Stack …
I have this proxy address: 125. 119. 175. 48:8909
How can I perform a HTTP request using cURL like curl, but specifying the proxy address of my network?
ivanleoncz6, 3093 gold badges49 silver badges45 bronze badges
asked Feb 25 ’12 at 15:47
0
From man curl:
-x, –proxy <[protocol][user:password@]proxyhost[:port]>
Use the specified HTTP proxy.
If the port number is not specified, it is assumed at port 1080.
bgcode24. 8k30 gold badges95 silver badges158 bronze badges
answered Feb 25 ’12 at 15:51
Karl BarkerKarl Barker10. 3k3 gold badges18 silver badges24 bronze badges
5
General way:
export _proxy=
Then you can connect through proxy from (many) application.
And, as per comment below, for:
answered Feb 25 ’12 at 19:51
airweenairween5, 4001 gold badge12 silver badges17 bronze badges
8
The above solutions might not work with some curl versions I tried them for myself(curl 7. 22. 0). But what worked for me was:
curl -x proxy_server:proxy_port –proxy-user username:password -L url
Hope it solves the issue better!
answered Feb 11 ’14 at 10:14
AmarAmar1, 9721 gold badge11 silver badges15 bronze badges
2
Beware that if you are using a SOCKS proxy, instead of a HTTP/HTTPS proxy, you will need to use the –socks5 switch instead:
curl –socks5 125. 48:8909
You can also use –socks5-hostname instead of –socks5 to resolve DNS on the proxy side.
answered Aug 28 ’14 at 16:57
Filipe CorreiaFilipe Correia4, 7454 gold badges26 silver badges46 bronze badges
4
as an adition to airween, another good idea is to add this into your, so you’ll be able to switch from non proxied to proxied environment:
alias proxyon=”export _proxy=’YOURPROXY:YOURPORT’;export _proxy=’YOURPROXY:YOURPORT'”
alias proxyoff=”export _proxy=”;export _proxy=””
WHERE YOURPROXY:YOURPORT is exactly that, your ip and port proxy:-).
Then, simply doing
proxyon
your system will start to use the proxy, and just the opposite with:
proxyoff
answered May 22 ’14 at 8:03
Alejandro MorenoAlejandro Moreno5, 1622 gold badges29 silver badges28 bronze badges
1
use the following
curl -I -x 192. 168. X. X:XX
192. X:XX put your proxy server ip and port.
-v verbose mode it will give more details including headers and response.
answered Jan 29 ’15 at 16:48
13krn13krn5014 silver badges9 bronze badges
I like using this in order to get the IP under which I am seen
curl -x proxy_server:proxy_port && echo
Hope this helps someone.
answered Sep 4 ’17 at 14:11
bmetgebmetge3113 silver badges3 bronze badges
For curl you can configure proxy in your ~/ (_curlrc on Windows) file by adding proxy value, the syntax is:
proxy = username:password@proxy-host:port
answered Mar 5 ’16 at 14:16
kenorbkenorb125k69 gold badges606 silver badges641 bronze badges
C:\Desktop>curl –head –proxy 1. 1. 1:8080
Shree19. 9k28 gold badges89 silver badges134 bronze badges
answered Sep 6 ’19 at 9:52
Just summarizing all great mentioned answers:
curl -x
M. MashayeM. Mashaye87411 silver badges21 bronze badges
With a proxy with authentication I use:
curl -x
because, I don’t know why curl doesn’t use/catch [s]_proxy environment variables.
answered May 31 ’17 at 6:39
You don’t need to export the [s]_proxy shell variable if you’re just setting the proxy for a one off command. e. g.
_proxy= curl
That said, I’d prefer curl -x if I knew I was always going to use a proxy.
answered Mar 27 ’17 at 18:03
overthinkoverthink22. 9k3 gold badges63 silver badges68 bronze badges
sudo curl -x -fsSL
This worked perfectly for me, the error comes because curl need to set
the proxy
Remmember replace the proxy with your proxy, mine, “example” was
answered Jun 11 ’19 at 16:26
Depending on your workplace, you may also need to specify the -k or the –insecure option for curl in order to get past potential issues with CA certificates.
curl -x
curl -vv -k -x
iamtheexp01iamtheexp013, 2968 gold badges30 silver badges35 bronze badges
In case the proxy is using automatic proxy with PAC file. We can find the actual proxy from the javascript from the PAC URL.
And if the proxy needs authentication, we can first use a normal web-browser to access the website which will promote authentication dialog. After authentication, we can use wireshark to capture the package sends to the proxy server, from the package, we can get the auth token from header: Proxy-Authorization
Then we can set the _proxy environment variable and also include auth token in the header: Proxy-Authorization
export _proxy=proxyserver:port
curl -H “Proxy-Authorization: xxxx” targetURL
answered Jun 24 ’17 at 5:49
Jianwu ChenJianwu Chen3, 8943 gold badges21 silver badges28 bronze badges
Not the answer you’re looking for? Browse other questions tagged linux curl proxy or ask your own question.
How to use curl command with proxy username/password on …
My sysadmin provided me the following proxy details:
IP: 202. 54. 1. 1
Port: 3128
Username: foo
Password: bar
The settings worked perfectly with Google Chrome and Firefox browser. How do I use it with the curl command? How do I tell the curl command to use my proxy settings from Google Chrome browser?
Many Linux and Unix command line tools such as curl command, wget command, lynx command, and others; use the environment variable called _proxy, _proxy, ftp_proxy to find the proxy details. It allows you to connect text based session and applications via the proxy server with or without a userame/password. This page shows how to perform HTTP/HTTPS requests with cURL cli using PROXY server.
Unix and Linux curl command with proxy syntax
The syntax is:
## Set the proxy address of your uni/company/vpn network ##
export _proxy=your-ip-address:port/
## _proxy with username and password
export _proxy=user:password@your-proxy-ip-address:port/
## HTTPS version ##
Another option is to pass the -x option to the curl command. To use the specified proxy:
curl -x <[protocol][user:password@]proxyhost[:port]> url
–proxy <[protocol][user:password@]proxyhost[:port]> url
–proxy user:password@Your-Ip-Here:Port url
-x user:password@Your-Ip-Here:Port url
Linux use curl command with proxy
First set the _proxy:
## proxy server, 202. 1, port: 3128, user: foo, password: bar ##
export _proxy=foo:bar@202. 1:3128/
export _proxy=$_proxy
## Use the curl command ##
curl -I curl -v -I
Sample outputs:
* Rebuilt URL to: * Trying 202. 1…
* Connected to 1202. 1 (202. 1) port 3128 (#0)
* Proxy auth using Basic with user ‘foo’
> HEAD HTTP/1. 1
> Host: > Proxy-Authorization: Basic x9VuUml2xm0vdg93MtIz
> User-Agent: curl/7. 43. 0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1. 1 200 OK
HTTP/1. 1 200 OK
< Server: nginx
Server: nginx
< Date: Sun, 17 Jan 2016 11:49:21 GMT
Date: Sun, 17 Jan 2016 11:49:21 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Whom: Dyno-l1-com-cyber
X-Whom: Dyno-l1-com-cyber
< Vary: Cookie
Vary: Cookie
< Link: <>; rel=”
Link: <>; rel=”
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< X-Cache: MISS from server1
X-Cache: MISS from server1
< X-Cache-Lookup: MISS from server1:3128
X-Cache-Lookup: MISS from server1:3128
< Connection: keep-alive
Connection: keep-alive
<
* Connection #0 to host 10. 12. 249. 194 left intact
In this example, I’m downloading a pdf file:
$ export _proxy="vivek:myPasswordHere@10. 194:3128/"
$ curl -v -O OR use the -x option:
curl -x 'vivek:myPasswordHere@10. 194:3128' -v -O Sample outputs:
Fig. 01: curl in action (click to enlarge)
How to use the specified proxy server with curl on Unix
$ curl -x prox_server_vpn:3128/ -I How to use socks protocol?
The syntax is same:
curl -x socks5[user:password@]proxyhost[:port]/ url
curl --socks5 192. 168. 254:3099
How do I configure and setup curl to permanently use a proxy connection?
Update/edit your ~/ file using a text editor such as vim:
$ vi ~/
Append the following:
Patreon supporters only guides
proxy =
proxy-user = "foo:bar"
Save and close the file. Another option is create a bash shell alias in your ~/ file:
## alias for curl command
## set proxy-server and port, the syntax is
## alias curl="curl -x {your_proxy_host}:{proxy_port}"
alias curl="curl -x "
Remember, the proxy string can be specified with a protocol prefix to specify alternative proxy protocols. Use socks4, socks4a, socks5 or socks5h to request the specific SOCKS version to be used. No protocol specified, and all others will be treated as HTTP proxies. If the port number is not specified in the proxy string, it is assumed to be 1080. The -x option overrides existing environment variables that set the proxy to use. If there’s an environment variable setting a proxy, you can set proxy to “” to override it. See curl command man page here for more info.
ADVERTISEMENT
CategoryList of Unix and Linux commandsDocumentationhelp • mandb • man • pinfoDisk space analyzersdf • duf • ncdu • pydfFile Managementcat • cp • less • mkdir • more • treeFirewallAlpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16. 04 • Ubuntu 18. 04 • Ubuntu 20. 04Linux Desktop AppsSkype • Spotify • VLC 3Modern utilitiesbat • exaNetwork UtilitiesNetHogs • dig • host • ip • nmapOpenVPNCentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18. 04Package Managerapk • aptProcesses Managementbg • chroot • cron • disown • fg • glances • gtop • jobs • killall • kill • pidof • pstree • pwdx • time • vtopSearchingag • grep • whereis • whichShell builtinscompgen • echo • printfText processingcut • revUser Informationgroups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • wWireGuard VPNAlpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20. 04
Frequently Asked Questions about curl proxy configuration
How do I use a proxy to curl?
On Curl for example you can set the proxy using the –proxy flag:curl http://example.com –proxy 127.0.0.1:8080. Sh.proxy = 127.0.0.1:8080.http_proxy = http://127.0.0.1:8080.Apr 17, 2018
What is curl proxy?
An HTTP proxy is a proxy that the client speaks HTTP with to get the transfer done. curl will, by default, assume that a host you point out with -x or –proxy is an HTTP proxy, and unless you also specify a port number it will default to port 1080 (and the reason for that particular port number is purely historical).
Does curl use system proxy?
The client curl (naturally) uses the library libcurl under the hood. libcurl respects the proxy environment variables named http_proxy, ftp_proxy, sftp_proxy etc. If set, libcurl will use the specified proxy for that URL scheme. So for a “FTP://” URL, the ftp_proxy is considered.Jan 24, 2018