Python Proxy

How to set an HTTP proxy in Python 2.7? – Stack Overflow

I am trying to run a script that installs pip: and am getting a connection timeout due to my network being behind an HTTP proxy. Is there some way I could configure an HTTP proxy in my Python 2. 7 installation to be able to install what I am trying to install?
Note: I am using Windows. Below is the error I am getting:
Downloading/unpacking pip
Cannot fetch index base URL Could not find any downloads that satisfy the requirement pip
No distributions at all found for pip
asked Jul 30 ’12 at 17:53
RolandoRolando46. 6k85 gold badges237 silver badges363 bronze badges
It looks like has been updated to use the environment variables _proxy and _proxy.
set _proxy=set _proxy=python
Linux/OS X:
export _proxy=export _proxy=sudo -E python
However if this still doesn’t work for you, you can always install pip through a proxy using setuptools’ easy_install by setting the same environment variables.
set _proxy=set _proxy=easy_install pip
export _proxy=export _proxy=sudo -E easy_install pip
Then once it’s installed, use:
pip install –proxy=”user:password@server:port” packagename
From the pip man page:
Have pip use a proxy server to access sites. This can be specified
using “” notation. If the password
is left out, pip will ask for it.
answered Jul 30 ’12 at 18:03
Ben BurnsBen Burns14. 4k3 gold badges29 silver badges55 bronze badges
On my network just setting _proxy didn’t work for me. The following points were relevant.
1 Setting _proxy for your user wont be preserved when you execute sudo – to preserve it, do:
sudo -E yourcommand
I got my install working by first installing cntlm local proxy. The instructions here is succinct:
Instead of student number, you’d put your domain username
2 To use the cntlm local proxy, exec:
pip install –proxy localhost:3128 pygments
answered Sep 26 ’12 at 9:36
eugenevdeugenevd7336 silver badges20 bronze badges
You can install pip (or any other package) with easy_install almost as described in the first answer. However you will need a HTTPS proxy, too. The full sequence of commands is:
You might also want to add a port to the proxy, such as {s}_proxy=
answered Oct 7 ’13 at 9:33
Adrian WAdrian W3, 49310 gold badges31 silver badges37 bronze badges
For installing pip with behind a proxy I went with the steps below. My server was even behind a jump server.
From the jump server:
ssh -R 18080:proxy-server:8080 my-python-server
On the “python-server”
export _proxy=localhost:18080; export _proxy=localhost:18080; export ftp_proxy=$_proxy
answered May 21 ’14 at 14:25
sastorslsastorsl1, 64313 silver badges16 bronze badges
cd C:\Python34\Scripts
install PackageName
answered Jan 16 ’16 at 5:11
Not the answer you’re looking for? Browse other questions tagged python or ask your own question.
Learn How To Configure And Utilize Proxies With Python - Zyte

Learn How To Configure And Utilize Proxies With Python – Zyte

Sending HTTP requests in Python is not necessarily easy. We have built-in modules like urllib, urllib2 to deal with HTTP requests. Also, we have third-party tools like Requests. Many developers use Requests because it is high level and designed to make it extremely easy to send HTTP requests.
But choosing the tool which is most suitable for your needs is just one thing. In the web scraping world, there are many obstacles we need to overcome. One huge challenge is when your scraper gets blocked. To solve this problem, you need to use proxies. In this article, I’m going to show you how to utilize proxies when using the Requests module so your scraper will not get banned.
Requests and proxies
In this part, we’re going to cover how to configure proxies in Requests. To get started we need a working proxy and a URL we want to send the request to.
Basic usage
import requests
proxies = {
“”: “,
“”: “, }
r = (“, proxies=proxies)
The proxies dictionary must follow this scheme. It is not enough to define only the proxy address and port. You also need to specify the protocol. You can use the same proxy for multiple protocols. If you need authentication use this syntax for your proxy:
user:pass@10. 10. 10:8000
Environment variables
In the above example, you can define proxies for each individual request. If you don’t need this kind of customization you can just set these environment variables:
export HTTP_PROXY=”
export HTTPS_PROXY=”
This way you don’t need to define any proxies in your code. Just make the request and it will work.
Proxy with session
Sometimes you need to create a session and use a proxy at the same time to request a page. In this case, you first have to create a new session object and add proxies to it then finally send the request through the session object:
s = ssion()
oxies = {
r = (“)
IP rotating
As discussed earlier, a common problem that we encounter while extracting data from the web is that our scraper gets blocked. It is frustrating because if we can’t even reach the website we won’t be able to scrape it either. The solution for this is to use some kind of proxy or rather multiple rotating proxies. A proxy solution will let us get around the IP ban.
To be able to rotate IPs, we first need to have a pool of IP addresses. We can use free proxies that we can find on the internet or we can use commercial solutions for this. Be aware, that if your product/service relies on scraped data a free proxy solution will probably not be enough for your needs. If a high success rate and data quality are important for you, you should choose a paid proxy solution like Zyte Smart Proxy Manager (formerly Crawlera).
IP rotation with requests
So let’s say we have a list of proxies. Something like this:
ip_addresses = [“85. 237. 57. 198:44959”, “116. 0. 2. 94:43379”, “186. 86. 247. 169:39168”, “185. 132. 179. 112:1080”, “190. 61. 44. 86:9991”]
Then, we can randomly pick a proxy to use for our request. If the proxy works properly we can access the given site. If there’s a connection error we might want to delete this proxy from the list and retry the same URL with another proxy.
proxy_index = random. randint(0, len(ip_addresses) – 1)
proxy = {“”: ip_addresses(proxy_index), “”: ip_addresses(proxy_index)}
(url, proxies=proxies)
# implement here what to do when there’s a connection error
# for example: remove the used proxy from the pool and retry the request using another one
There are multiple ways you can handle connection errors. Because sometimes the proxy that you are trying to use is just simply banned. In this case, there’s not much you can do about it other than removing it from the pool and retrying using another proxy. But other times if it isn’t banned you just have to wait a little bit before using the same proxy again.
Implementing your own smart proxy solution which finds the best way to deal with errors is very hard to do. That’s why you should consider using a managed solution, like Zyte Smart Proxy Manager (formerly Crawlera), to avoid all the unnecessary pains with proxies.
Using Zyte Proxy Manager (formerly Crawlera) with requests
As a closing note, I want to show you how to solve proxy issues in the easiest way with Zyte Proxy Manager.
url = ”
proxy_host = “”
proxy_port = “8010”
proxy_auth = “:”
“”: “{}@{}:{}/”(proxy_auth, proxy_host, proxy_port),
“”: “{}@{}:{}/”(proxy_auth, proxy_host, proxy_port)}
r = (url, proxies=proxies, verify=False)
What does this piece of code do? It sends a successful HTTP request. When you use Zyte Proxy Manager, you don’t need to deal with proxy rotation manually. Everything is taken care of internally.
If you find that managing proxies on your own is too complex and you’re looking for an easy solution, give Zyte Smart Proxy Manager (formerly Crawlera) a go. It has a free trial!
qwj/python-proxy - GitHub

qwj/python-proxy – GitHub

HTTP/HTTP2/HTTP3/Socks4/Socks5/Shadowsocks/SSH/Redirect/Pf/QUIC TCP/UDP asynchronous tunnel proxy implemented in Python3 asyncio.
$ pip3 install pproxy
Successfully installed pproxy-1. 9. 5
$ pproxy
Serving on:8080 by, socks4, socks5
$ pproxy -l sschacha20:abc@:8080
Serving on:8080 by ss (chacha20-py)
Optional: (better performance with C ciphers)
$ pip3 install pproxy[accelerated]
Successfully installed pycryptodome-3. 6. 4
Apply OS system-wide proxy: (MacOS, Windows)
$ pproxy -r sschacha20:abc@server_ip:8080 –sys -vv
System proxy setting -> socks5 localhost:8080
socks5::1:57345 -> ss server_ip:8080 ->
socks5::1:57345 -> ss server_ip:8080 ->….. (all local traffic log)……
Apply CLI proxy: (MacOS, Linux)
$ export _proxy=localhost:8080
Run With Docker
pproxy Docker container has both python3 (with Cryptodome for performance optimizations) and pypy versions available.
docker run -it -p 8080:8080 mosajjal/pproxy:latest -l:8080 -vv
docker run -it -p 8080:8080 mosajjal/pproxy:latest-pypy -l:8080 -vv
Lightweight single-thread asynchronous IO.
Pure python, no additional library required.
Proxy client/server for TCP/UDP.
Schedule (load balance) among remote servers.
Incoming traffic auto-detect.
Tunnel/jump/backward-jump support.
Unix domain socket support.
User/password authentication support.
Filter/block hostname by regex patterns.
SSL/TLS client/server support.
Shadowsocks OTA (One-Time-Auth), SSR plugins.
Statistics by bandwidth and traffic.
PAC support for javascript configuration.
Iptables/Pf NAT redirect packet tunnel.
System proxy auto-setting support.
Client/Server API provided.
TCP server
TCP client
UDP server
UDP client

(get, post, etc)
(as client)
v2 (connect)
v3 (connect)
✔ by UDP
✔ udp-only
socks5 over TLS
shadowsocks aead
ssh tunnel
iptables nat
pfctl nat (macos)
(raw socket)
(simple tunnel)
xxx over TLS
Scheduling Algorithms
-s fa
-s rr
-s rc
-s lc
pycryptodome is an optional library to enable faster (C version) cipher. pproxy has many built-in pure python ciphers. They are lightweight and stable, but slower than C ciphers. After speedup with PyPy, pure python ciphers can get similar performance as C version. If the performance is important and don’t have PyPy, install pycryptodome instead.
asyncssh is an optional library to enable ssh tunnel client support.
These are some performance benchmarks between Python and C ciphers (dataset: 8M):
0. 64 secs
chacha20-py (pypy3)
1. 32 secs
48. 86 secs
PyPy3 Quickstart:
$ pypy3 -m ensurepip
$ pypy3 -m pip install asyncio pproxy
$ pproxy -h
usage: pproxy [-h] [-l LISTEN] [-r RSERVER] [-ul ULISTEN] [-ur URSERVER]
[-b BLOCK] [-a ALIVED] [-v] [–ssl SSLFILE] [–pac PAC]
[–get GETS] [–sys] [–test TESTURL] [–version]
Proxy server that can tunnel among remote servers by regex rules. Supported
protocols:, socks4, socks5, shadowsocks, shadowsocksr, redirect, pf, tunnel
optional arguments:
-h, –help show this help message and exit
-l LISTEN tcp server uri (default: +socks4+socks5:8080/)
-r RSERVER tcp remote server uri (default: direct)
-ul ULISTEN udp server setting uri (default: none)
-ur URSERVER udp remote server uri (default: direct)
-b BLOCK block regex rules
-a ALIVED interval to check remote alive (default: no check)
-s {fa, rr, rc, lc} scheduling algorithm (default: first_available)
-v print verbose output
–ssl SSLFILE certfile[, keyfile] if server listen in ssl mode
–pac PAC PAC path
–get GETS custom {path, file}
–sys change system proxy setting (mac, windows)
–test TEST test this url for all remote proxies and exit
–version show program’s version number and exit
Online help: <>
URI Syntax
{scheme}[{cipher}@]{netloc}/[@{localbind}][, {plugins}][? {rules}][#{auth}]
Currently supported scheme:, socks, ss, ssl, secure. You can use + to link multiple protocols together.
protocol (CONNECT)
protocol (GET/POST)
socks4 protocol
socks5 protocol
shadowsocks protocol
shadowsocksr (SSR) protocol
trojan protocol
ssh client tunnel
redirect (iptables nat)
pfctl (macos pf nat)
unsecured ssl/tls (no cert)
secured ssl/tls (cert)
raw connection
websocket connection
echo-back service
direct connection
“” accepts GET/POST/CONNECT as server, sends CONNECT as client. “only” sends “GET/POST” as client, works only on traffic.
Valid schemes:, +socks4+socks5, +ssl, ss+secure, +socks5+ss
Invalid schemes: ssl, secure
Cipher’s format: “cipher_name:cipher_key”. Cipher can be base64-encoded. So cipher string with “YWVzLTEyOC1nY206dGVzdA==” is equal to “aes-128-gcm:test”.
Full cipher support list:
Key Length
IV Length
Score (0-5)
0 (lowest)
0. 5
5 (highest)
4. 5
3. 5
2. 5
1. 5
pproxy ciphers have pure python implementations. Program will switch to C cipher if there is C implementation available within pycryptodome. Otherwise, use pure python cipher.
AEAD ciphers use additional payload after each packet. The underlying protocol is different. Specifications: AEAD.
Some pure python ciphers (aes-256-cfb1-py) is quite slow, and is not recommended to use without PyPy speedup. Try install pycryptodome and use C version cipher instead.
To enable OTA encryption with shadowsocks, add ‘! ‘ immediately after cipher name.
It can be “hostname:port” or “/unix_domain_socket”. If the hostname is empty, server will listen on all interfaces.
Valid netloc: localhost:8080, 0. 0. 0:8123, /tmp/domain_socket, :8123
It can be “@in” or @ipv4_address or @ipv6_address
Valid localbind: @in, @192. 168. 1. 15, @::1
It can be multiple plugins joined by “, “. Supported plugins: plain, origin, _simple, tls1. 2_ticket_auth, verify_simple, verify_deflate
Valid plugins: /, tls1. 2_ticket_auth, verify_simple
The filename that contains regex rules
The username, colon ‘:’, and the password
URIs can be joined by “__” to indicate tunneling by jump. For example, ss1. 2. 3. 4:1324__ make remote connection to the first shadowsocks proxy server, and then jump to the second proxy server.
Client API
TCP Client API
import asyncio, pproxy
async def test_tcp(proxy_uri):
conn = nnection(proxy_uri)
reader, writer = await p_connect(”, 80)
(b’GET / HTTP/1. 1\r\n\r\n’)
data = await (1024*16)
UDP Client API
async def test_udp(proxy_uri):
answer = ()
await conn. udp_sendto(‘8. 8. 8′, 53, b’hello the world’, t_result)
await answer
Server API
Server API example:
import asyncio
import pproxy
server = (‘ss0. 0:1234’)
remote = nnection(‘ss1. 4:5678’)
args = dict( rserver = [remote],
verbose = print)
loop = t_event_loop()
handler = n_until_complete(art_server(args))
except KeyboardInterrupt:
print(‘exit! ‘)
n_until_complete(handler. wait_closed())
Regex rule
Define regex file “rules” as follow:
#google domains
(? :. +\. )? google. *\
(? :. )? gstatic\
(? :. )? gmail\
(? :. )? ntp\
(? :. )? glpals\
(? :. )? akamai. )? ggpht\
(? :. )? android\
(? :. )? gvt1\
(? :. )? youtube. )? ytimg\
(? :. )? goo\
(? :. )? youtu\
(? :. )? google\.. +
Then start pproxy
$ pproxy -r -vv
Serving on:8080 by, socks4, socks5::1:57768 -> ->::1:57772 -> socks4::1:57770 -> -> pproxy will serve incoming traffic by /socks4/socks5 auto-detect protocol, redirect all google traffic to proxy, and visit all other traffic directly from local.
Use cipher
Add cipher encryption to make sure data can’t be intercepted. Run pproxy locally as:
$ pproxy -l ss:8888 -r ss -vv
Next, run remotely on server “”. The base64 encoded string of “chacha20:cipher_key” is also supported:
$ pproxy -l sschacha20:cipher_key@:12345
The same as:
$ pproxy -l ssY2hhY2hhMjA6Y2lwaGVyX2tleQ==@:12345
The traffic between local and is encrypted by stream cipher Chacha20 with secret key “cipher_key”.
Unix domain socket
A more complex example:
$ pproxy -l sssalsa20! :complex_cipher_key@/tmp/pproxy_socket -r +ssl
pproxy listen on the unix domain socket “/tmp/pproxy_socket” with cipher “salsa20” and key “complex_cipher_key”. OTA packet protocol is enabled by adding! after cipher name. The traffic is tunneled to remote proxy with simple authentication.
SSL/TLS server
If you want to listen in SSL/TLS, you must specify ssl certificate and private key files by parameter “–ssl”:
$ pproxy -l +ssl0. 0:443 -l –ssl, –pac /autopac
pproxy listen on both 80 HTTP and 443 HTTPS ports, use the specified SSL/TLS certificate and private key files. The “–pac” enable PAC feature, so you can put ” path in your device’s auto-configure url.
Simple guide for generating self-signed ssl certificates:
$ openssl genrsa -des3 -out 1024
$ openssl req -new -key -out
$ cp
$ openssl rsa -in -out
$ openssl x509 -req -days 365 -in -signkey -out
SSR plugins
ShadowsocksR example with plugin “tls1. 2_ticket_auth” to emulate common tls traffic:
$ pproxy -l ssrchacha20:mypass@0. 0:443/, tls1. 2_ticket_auth, verify_simple
Local bind ip
If you want to route the traffic by different local bind, use the @localbind URI syntax. For example, server has three ip interfaces: 192. 15, 111. 1, 112. You want to route traffic matched by “rule1” to 111. 2 and traffic matched by “rule2” to 222. 2, and the remaining traffic directly:
$ pproxy -l ss:8000/@in -r ss111. 2:8000/@111. 1? rule1 -r ss222. 2:8000/@222. 1? rule2
Redirect/Pf protocol
IPTable NAT redirect example (Ubuntu):
$ sudo iptables -t nat -A OUTPUT -p tcp –dport 80 -j REDIRECT –to-ports 5555
$ pproxy -l redir:5555 -r remote__server:3128 -vv
The above example illustrates how to redirect all local output tcp traffic with destination port 80 to localhost port 5555 listened by pproxy, and then tunnel the traffic to remote proxy.
PF redirect example (MacOS):
$ sudo pfctl -ef /dev/stdin
rdr pass on lo0 inet proto tcp from any to any port 80 -> 127. 1 port 8080
pass out on en0 route-to lo0 inet proto tcp from any to any port 80 keep state
$ sudo pproxy -l pf:8080 -r socks5remote_socks5_server:1324 -vv
Make sure pproxy runs in root mode (sudo), otherwise it cannot redirect pf packet.
Multiple jumps example
$ pproxy -r server1__ssserver2__socksserver3
pproxy will connect to server1 first, tell server1 connect to server2, and tell server2 connect to server3, and make real traffic by server3.
Raw connection tunnel
TCP raw connection tunnel example:
$ pproxy -l tunnel{}:80
$ curl -H “Host: ” localhost
UDP dns tunnel example:
$ pproxy -ul tunnel{8. 8}:53
$ nslookup localhost
UDP more complicated example
Run the shadowsocks udp proxy on remote machine:
$ pproxy -ul ssremote_server:13245
Run the commands on local machine:
$ pproxy -ul tunnel{8. 8}:53 -ur ssremote_server:13245 -vv
UDP tunnel 127. 1:60573 -> ss remote_server:13245 -> 8. 8:53
UDP tunnel 127. 1:60574 -> ss remote_server:13245 -> 8. 8:53…
Load balance example
Specify multiple -r server, and a scheduling algorithm (rr = round_robin, rc = random_choice, lc = least_connection):
$ pproxy -r server1 -r ssserver2 -r socks5server3 -s rr -vv::1:42356 -> server1 ->::1:42357 -> ss server2 ->::1:42358 -> socks5 server3 ->::1:42359 -> server1 ->…
$ pproxy -ul tunnel:53 -ur tunnel8. 8:53 -ur tunnel8. 4. 4:53 -s rc -vv
UDP tunnel::1:35378 -> tunnel 8. 8:53
UDP tunnel::1:35378 -> tunnel 8. 4:53…
WebSocket example
WebSocket protocol is similar to Tunnel protocol. It is raw and doesn’t support any proxy function. It can connect to other proxy like Tunnel protocol.
First run pproxy on remote machine:
$ pproxy -l ws:80 -r tunnel/tmp/myproxy -v
$ pproxy -l sschacha20:abc@/tmp/myproxy -v
Run pproxy on local machine:
$ pproxy -l tunnel:1234 -r wsremote_ip:80 -vv
Then port:1234 on local machine is connected to the /tmp/myproxy on remote machine by WebSocket tunnel. You can specify any proxy protocol details on /tmp/myproxy.
It is a good practice to use some CDN in the middle of local/remote machines. CDN with WebSocket support can hide remote machine’s real IP from public.
Backward proxy
Sometimes, the proxy server hides behind an NAT router and doesn’t have a public ip. The client side has a public ip “client_ip”. Backward proxy feature enables the server to connect backward to client and wait for proxy requests.
Run pproxy client as follows:
$ pproxy -l:8080 -r +in:8081 -v
Run pproxy server as follows:
$ pproxy -l +inclient_ip:8081
Server connects to client_ip:8081 and waits for client proxy requests. The protocol specified is just an example. It can be any protocol and cipher pproxy supports. The scheme “in” should exist in URI to inform pproxy that it is a backward proxy.
$ pproxy -l +injumpserver__client_ip:8081
It is a complicated example. Server connects to client_ip:8081 by jump jumpserver. The backward proxy works through jumps.
SSH client tunnel
SSH client tunnel support is enabled by installing additional library asyncssh. After “pip3 install asyncssh”, you can specify “ssh” as scheme to proxy via ssh client tunnel.
$ pproxy -l:8080 -r ssh
If a client private key is used to authenticate, put double colon “::” between login and private key path.
SSH connection known_hosts feature is disabled by default.
SSH jump
SSH jump is supported by using “__” concatenation
$ pproxy -r sshserver1__sshserver2__sshserver3
First connection to server1 is made. Second, ssh connection to server2 is made from server1. Finally, connect to server3, and use server3 for proxying traffic.
SSH remote forward
$ pproxy -l sshserver__tunnel0. 0:1234 -r tunnel127. 1:1234
TCP:1234 on remote server is forwarded to 127. 1:1234 on local server
$ pproxy -l sshserver1__sshserver2__ss0. 0:1234 -r ssserver3:1234
It is a complicated example. SSH server2 is jumped from SSH server1, and ss0. 0:1234 on server2 is listened. Traffic is forwarded to ssserver3:1234.
Trojan protocol example
Normally trojan should be used together with ssl. You should specify the SSL crt/key file for ssl usage. A typical trojan server would be:
$ pproxy –ssl, -l trojan+tunnel{localhost:80}+ssl:443#yourpassword -vv
If trojan password doesn’t match, the tunnal{localhost:80} will be switched to. It looks exactly the same as a common HTTPS website.
QUIC protocol example
QUIC is a UDP stream protocol used in HTTP/3. Library aioquic is required if you want to proxy via QUIC.
QUIC is listened on UDP port, but can handle TCP or UDP traffic. If you want to handle TCP traffic, you should use “-l quic+” instead of “-ul quic+”.
$ pip3 install aioquic
$ pproxy –ssl, -l quic+:1234
On the client:
$ pproxy -r quic+server:1234
QUIC protocol can transfer a lot of TCP streams on one single UDP stream. If the connection number is hugh, QUIC can benefit by reducing TCP handshake time.
VPN Server Example
You can run VPN server simply by installing pvpn (python vpn), a lightweight VPN server with pproxy tunnel feature.
$ pip3 install pvpn
Successfully installed pvpn-0. 1
$ pvpn -wg 9999 -r remote_server:remote_port
Serving on UDP:500:4500…
Serving on UDP:9000 (WIREGUARD)…
TCP -> HTTP ->
python-vpn – VPN Server (IPSec, IKE, IKEv2, L2TP, WireGuard) in pure python
shadowproxy – Awesome python proxy implementation by guyingbo

Frequently Asked Questions about python proxy

What is proxy in Python?

Proxy is a structural design pattern that provides an object that acts as a substitute for a real service object used by a client. A proxy receives client requests, does some work (access control, caching, etc.) and then passes the request to a service object.

How do I make a proxy in Python?

Creating a Proxy Webserver in Python | Set 1Creating an incoming socket. We create a socket serverSocket in the __init__ method of the Server Class. This creates a socket for the incoming connections. … Accept client and process. This is the easiest yet the most important of all the steps. … Redirecting the traffic.Nov 15, 2017

How does Python connect to proxy server?

“how to connect to a proxy server using python” Code Answerimport requests.proxies = { “http”: “″,”https”: “”}r = requests. get(“”, proxies=proxies)More items…•Mar 6, 2020

Leave a Reply

Your email address will not be published. Required fields are marked *