performing HTTP requests with cURL (using PROXY) – Stack …
I have this proxy address: 125. 119. 175. 48:8909
How can I perform a HTTP request using cURL like curl, but specifying the proxy address of my network?
ivanleoncz6, 3093 gold badges49 silver badges45 bronze badges
asked Feb 25 ’12 at 15:47
0
From man curl:
-x, –proxy <[protocol][user:password@]proxyhost[:port]>
Use the specified HTTP proxy.
If the port number is not specified, it is assumed at port 1080.
bgcode24. 8k30 gold badges95 silver badges158 bronze badges
answered Feb 25 ’12 at 15:51
Karl BarkerKarl Barker10. 3k3 gold badges18 silver badges24 bronze badges
5
General way:
export _proxy=
Then you can connect through proxy from (many) application.
And, as per comment below, for:
answered Feb 25 ’12 at 19:51
airweenairween5, 4001 gold badge12 silver badges17 bronze badges
8
The above solutions might not work with some curl versions I tried them for myself(curl 7. 22. 0). But what worked for me was:
curl -x proxy_server:proxy_port –proxy-user username:password -L url
Hope it solves the issue better!
answered Feb 11 ’14 at 10:14
AmarAmar1, 9721 gold badge11 silver badges15 bronze badges
2
Beware that if you are using a SOCKS proxy, instead of a HTTP/HTTPS proxy, you will need to use the –socks5 switch instead:
curl –socks5 125. 48:8909
You can also use –socks5-hostname instead of –socks5 to resolve DNS on the proxy side.
answered Aug 28 ’14 at 16:57
Filipe CorreiaFilipe Correia4, 7454 gold badges26 silver badges46 bronze badges
4
as an adition to airween, another good idea is to add this into your, so you’ll be able to switch from non proxied to proxied environment:
alias proxyon=”export _proxy=’YOURPROXY:YOURPORT’;export _proxy=’YOURPROXY:YOURPORT'”
alias proxyoff=”export _proxy=”;export _proxy=””
WHERE YOURPROXY:YOURPORT is exactly that, your ip and port proxy:-).
Then, simply doing
proxyon
your system will start to use the proxy, and just the opposite with:
proxyoff
answered May 22 ’14 at 8:03
Alejandro MorenoAlejandro Moreno5, 1622 gold badges29 silver badges28 bronze badges
1
use the following
curl -I -x 192. 168. X. X:XX
192. X:XX put your proxy server ip and port.
-v verbose mode it will give more details including headers and response.
answered Jan 29 ’15 at 16:48
13krn13krn5014 silver badges9 bronze badges
I like using this in order to get the IP under which I am seen
curl -x proxy_server:proxy_port && echo
Hope this helps someone.
answered Sep 4 ’17 at 14:11
bmetgebmetge3113 silver badges3 bronze badges
For curl you can configure proxy in your ~/ (_curlrc on Windows) file by adding proxy value, the syntax is:
proxy = username:password@proxy-host:port
answered Mar 5 ’16 at 14:16
kenorbkenorb125k69 gold badges606 silver badges641 bronze badges
C:\Desktop>curl –head –proxy 1. 1. 1:8080
Shree19. 9k28 gold badges89 silver badges134 bronze badges
answered Sep 6 ’19 at 9:52
Just summarizing all great mentioned answers:
curl -x
M. MashayeM. Mashaye87411 silver badges21 bronze badges
With a proxy with authentication I use:
curl -x
because, I don’t know why curl doesn’t use/catch [s]_proxy environment variables.
answered May 31 ’17 at 6:39
You don’t need to export the [s]_proxy shell variable if you’re just setting the proxy for a one off command. e. g.
_proxy= curl
That said, I’d prefer curl -x if I knew I was always going to use a proxy.
answered Mar 27 ’17 at 18:03
overthinkoverthink22. 9k3 gold badges63 silver badges68 bronze badges
sudo curl -x -fsSL
This worked perfectly for me, the error comes because curl need to set
the proxy
Remmember replace the proxy with your proxy, mine, “example” was
answered Jun 11 ’19 at 16:26
Depending on your workplace, you may also need to specify the -k or the –insecure option for curl in order to get past potential issues with CA certificates.
curl -x
curl -vv -k -x
iamtheexp01iamtheexp013, 2968 gold badges30 silver badges35 bronze badges
In case the proxy is using automatic proxy with PAC file. We can find the actual proxy from the javascript from the PAC URL.
And if the proxy needs authentication, we can first use a normal web-browser to access the website which will promote authentication dialog. After authentication, we can use wireshark to capture the package sends to the proxy server, from the package, we can get the auth token from header: Proxy-Authorization
Then we can set the _proxy environment variable and also include auth token in the header: Proxy-Authorization
export _proxy=proxyserver:port
curl -H “Proxy-Authorization: xxxx” targetURL
answered Jun 24 ’17 at 5:49
Jianwu ChenJianwu Chen3, 8943 gold badges21 silver badges28 bronze badges
Not the answer you’re looking for? Browse other questions tagged linux curl proxy or ask your own question.
How to use cURL with proxy? – Blog | Oxylabs
This step-by-step guide will explain how to use cURL or simply, curl, with proxy servers. It covers all the aspects, beginning from installation to explaining various options to set the proxy.
We did not target any specific proxy service. Therefore this tutorial should work with all proxy servers. All you need to know is the server details and credentials.
This is a fairly technical tutorial and expects readers to have a basic understanding of what a proxy is. It would be especially interesting and useful for those starting with web scraping.
Navigation:
What is cURL?
Installation
What you need to connect to a proxy
Using cURL with HTTP/HTTPS proxy
Command line argument to set proxy in cURL
Using environment variables
Configure cURL to always use proxy
Ignore or override proxy for one request
Bonus tip – turning proxies off and on quickly
cURL socks proxy
Summary
cURL is a command line tool for sending and receiving data using the url. Let’s look at the simplest example of using curl. Open your terminal or command prompt and type in this command and press Enter:
curl This will get the HTML of the page and print it on the console.
curl -I
This will print the document information.
HTTP/1. 1 200 OK
Content-Type: text/html; charset=ISO-8859-1
The question “what is cURL? ” is also answered in one of our previous articles. We recommend reading it if you want to learn how it became such a universal asset.
cURL is provided with many Linux distributions and with MacOS. Now it is provided with Windows 10 as well.
If your Linux distribution is not provided with it, you can install it by running the install command. For example, on Ubuntu, open Terminal and run this command:
sudo apt install curl
If you are running an older version of Windows, or if you want to install an alternate version, you can download curl from the official download page.
Irrespective of which proxy service you use, you will need the following information to use a:
proxy server addressportprotocolusername (if authentication is required)password (if authentication is required)
In this tutorial, we are going to assume that the proxy server is 127. 0. 1, the port is 1234, the user name is user, and the password is pwd. We will look into multiple examples covering various protocols.
NOTE. If you are on a network that uses NTLM authentication, you can use the switch –proxy-ntlm while running curl. Similarly, –proxy-digest can be used for digest authentication. You can look at all the available options by running curl –help. This tutorial will have examples for the scenario when a username and password has to be specified.
The next section will cover the first curl proxy scenario, which happens to be the most common one – HTTP and HTTPS proxy with curl.
If you recall, we looked at using curl without proxy like this:
curl This particular website is especially useful for testing out proxies as the output of this page is the origin IP address. If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address.
There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.
NOTE. All the command line options, or switches, are case sensitive. For example, -f instructs curl to fail silently, while -F denotes a form to be submitted.
Open terminal and type the following command, and press Enter:
curl –help
The output is going to be a huge list of options. One of them is going to look like this:
-x, –proxy [protocol]host[:port]
Note that x is small, and it is case-sensitive. The proxy details can be supplied using -x or –proxy switch. Both mean the same thing. Bot of the curl with proxy commands are same:
curl -x “user:pwd@127. 1:1234″ ”
or
curl –proxy “user:pwd@127. 1:1234″ ”
NOTE. If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.
curl –proxy “user:pwd@127. 1:1234″ ” -k
You may have noticed that both the proxy url and target url are surrounded in double quotes. This is a recommended practice to handle special characters in the url.
Another interesting thing to note here is that the default proxy protocol is. Thus, following two commands will do exactly the same:
Another way to use proxy with curl is to set the environment variables _proxy and _proxy.
Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc file.
If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.
_proxy – the proxy will be used to access addresses that use protocol
Simply set the variables _proxy to proxy address and _proxy to set proxy address. Open terminal and run these two commands.
export _proxy=”user:pwd@127. 1:1234″
After running these two commands, run curl normally.
curl ”
If you see SSL Certificate errors, add -k to ignore these errors.
Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:
unset _proxy
See the next section to set default proxy only for curl and not system wide.
If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.
For Linux and MacOS, open terminal and navigate to your home directory. If there is already a file, open it. If there is none, create a new file. Here are the set of commands that can be run:
cd ~
nano
In this file, add this line:
proxy=”user:pwd@127. 1:1234″
Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from file.
On Windows, the file is named _curlrc. This file can be placed in the%APPDATA% directory.
To find the exact path of%APPDATA%, open command prompt and run the following command:
echo%APPDATA%
This directory will be something like C:\Users\
This works exactly the same way in Linux, MacOS, and Windows.
If the proxy is set globally, or by modifying the file, this can still be overridden to set another proxy or even bypass it.
To override proxy for one request, set the new proxy using -x or –proxy switch as usual:
curl –proxy “user:pwd@1. 1:8090″ ”
If you want to bypass proxy altogether for a request, you can pass –noproxy followed by “*”. This instructs curl to not use proxy for all URLs.
curl –noproxy “*” ”
If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.
This tip is dedicated only for advanced users. If you do not know what a file is, you may skip this section.
You can create an alias in your file to set proxies and unset proxies. For example, open file using any editor and add these lines:
alias proxyon=”export _proxy=’ user:pwd@127. 1:1234′;export _proxy=’ user:pwd@127. 1:1234′”
alias proxyoff=”unset _proxy;unset _proxy”
After adding these lines, save the and update the shell to read this To do this, run this this command in the terminal:. ~/
Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:
proxyon
proxyoff
If the proxy server is using socks protocol, the syntax remains the same:
curl -x “socks5user:pwd@127. 1:1234″ ”
Similarly, socks4, socks4a, socks5 or socks5h can be used depending on the socks version.
Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.
curl –socks5 “127. 1:1234″ ” –proxy-user user:pwd
Again, –socks4, –socks4a or –socks5 can be used, depending on the version.
cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.
You can learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.
Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you’ll probably find her just chillin’ while listening to her favorite music or playing board games with friends.
All information on Oxylabs Blog is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.
How do I use Curl with a proxy? – ReqBin
What is Curl? Curl is a command-line tool that allows users to transfer data over the network. Curl supports over 25+ protocols, including HTTP, HTTPS, FTP, and SFTP. When it comes to debugging network requests and APIs calls, Curl is one of the best tools out there, and it’s free and open-source. Curl works on Linux, Mac, and Windows. What is a Proxy Server? A proxy server is an intermediate server (software or hardware) between the user (browser, mobile app, etc. ) and the website and transfers traffic between them. Proxies separate end-user clients from the websites they browse and provide multiple levels of functionality and security. For example, proxy servers can compress and encrypt network traffic, prevent intrusion into a local network from the Internet, and monitor employee Internet use during working hours. The word “proxy” means “to act on behalf of another, ” and the proxy server acts on behalf of the end-user. When using proxy servers, all requests to the Internet first go to the proxy server, which evaluates the request, applies a set of rules to it, and then, if necessary, forwards the request to the Internet. Likewise, server responses are first returned to the proxy, which evaluates and processes them and is only sent to the end-user. In some cases, a chain of proxy servers is used to provide greater anonymity on the Internet. Does Curl support proxies? Yes, Curl supports proxies via the -x or –proxy command line option. The general form of the Curl command to send a request through a proxy server is as follows: Curl Proxy Syntax curl -x “[protocol][host][:port]” [URL] [options] It is recommended that you enclose the proxy server address in double quotes to handle special characters in the URL correctly. Note that the default proxy protocol for Curl is HTTP. Passing Proxy Address to Curl via Environment Variables Another way to pass the proxy address to Curl is to set the _proxy and _proxy environment variables. Using these environment variables will have the same effect as using the -x –proxy command line option. Just keep in mind that command line options are more powerful and override parameters set via environment variables. Curl Proxy Settings via Environment Variables export _proxy=”[protocol][host][:port]”
export _proxy=”[protocol][host][:port]” To disable global proxy settings, unset these two environment variables. unset _proxy
unset _proxy What is an Environment Variable? An environment variable is a key=value pair that is set at the system level and can affect the behavior of all running processes on the computer. Environment variables are the part of the operating system in which a process runs. Environment variables were first introduced in 1979 on Unix and are now supported by all operating systems, including Linux, macOS, and Windows. How to bypass SSL certificate errors when using Curl Proxy? If Curl gets an SSL certificate error, it will automatically block the request. For debugging purposes, to allow insecure connections to the server, you can tell Curl to ignore SSL certificate errors by adding the -k command-line option to the curl request. Allow Insecure SSL Connections for Curl Proxy Requests curl -x “[protocol][host][:port]” -k [URL] How to make Curl use a proxy via the file? You can avoid passing the proxy address on the Curl command line every time by creating a Curl config file () and setting the proxy address in it. Create a and enter the proxy address like this: Set Сurl Proxy Address in # — Proxy example file —
proxy = “[protocol][host][:port]” How to ignore proxies for a single Curl request? Suppose the proxy is set globally via an environment variable or file. In that case, you can still bypass it for specific Curl requests by using the –noproxy “*” command-line parameter in the Curl request. Disable Proxy for Specific Curl Requests curl –noproxy “*” What is a SOCKS Proxy? SOCKS (Socket Secure) is a network communication protocol that directs any traffic generated by an application to a real server on behalf of the client. A SOCKS proxy server creates a Transmission Control Protocol (TCP) connection with another server and exchanges network packets between the client and the server. SOCKS can handle several types of requests and transparently transfer data for HTTP, HTTPS, POP3, SMTP, FTP, and SFT streams. There are only two versions of the SOCKS protocol: SOCKS4 and SOCKs5. The main differences between SOCKs5 and SOCKS4: SOCKS5 supports multiple authentication methods, SOCKS4 does not support authentication; SOCKS5 supports UDP proxy, SOCKS4 does not support UDP proxy; SOCKS5 is more secure because it uses an authenticated TCP connections and SSH encrypted tunnels; Syntax: Curl SOCKS Proxy Syntax curl -x “[socks5][host][:port]” [URL] [options] For Curl SOCKS proxies, you can use the –socks5 command line parameter instead of -x. Curl SOCKS Proxy Syntax curl –socks5 “[host][:port]” [URL] [options] What is Proxy-Authorization? Proxy authorization is a built-in HTTP mechanism called proxy authentication that prevents unauthorized use of the proxy server and blocks requests from clients to the proxy until the user provides valid credentials to access the proxy. Proxy server authorization is accomplished by sending the Proxy-Authorization HTTP request header from the client to the proxy server. Proxy Authorization Syntax Proxy-Authorization: [type] [credentials]
Proxy-Authorization: Basic bG9naW46cGFzc3dvcmQ=
How to specify proxy username and password for Curl? If your proxy requires user authentication, you can pass the username and password to Curl with the -U or –proxy-user command-line switch. Curl Proxy Authentication curl -x “[protocol][host][:port]” [URL] -U login:password See also List of popular Curl flags How do I use Curl with SSL connections? 12 Essential Curl Commands for Linux, Windows and macOS
Frequently Asked Questions about curl use proxy
Can curl use proxy?
To use a proxy with Curl, you must pass the required proxy address using the -x (or –proxy) command-line option and proxy credentials using the -U (or –proxy-user) command-line switch. Proxy credentials may also be passed in the proxy string and will be URL decoded by Curl.Jul 4, 2021
How do I use a proxy to curl?
On Curl for example you can set the proxy using the –proxy flag:curl http://example.com –proxy 127.0.0.1:8080. Sh.proxy = 127.0.0.1:8080.http_proxy = http://127.0.0.1:8080.Apr 17, 2018
What is curl proxy?
An HTTP proxy is a proxy that the client speaks HTTP with to get the transfer done. curl will, by default, assume that a host you point out with -x or –proxy is an HTTP proxy, and unless you also specify a port number it will default to port 1080 (and the reason for that particular port number is purely historical).