How to make python Requests work via socks proxy – Stack …
I’m using the great Requests library in my Python script:
import requests
r = (“”)
print
I would like to use socks proxy. But Requests only supports HTTP proxy now.
How can I do that?
asked Sep 26 ’12 at 12:02
The modern way:
pip install -U requests[socks]
then
resp = (”,
proxies=dict(=’socks5user:pass@host:port’,
=’socks5user:pass@host:port’))
answered Mar 27 ’13 at 14:20
dvskadvska1, 8791 gold badge15 silver badges14 bronze badges
9
As of requests version 2. 10. 0, released on 2016-04-29, requests supports SOCKS.
It requires PySocks, which can be installed with pip install pysocks.
Example usage:
proxies = {”: “socks5myproxy:9191″}
(”, proxies=proxies)
Mark Amery117k61 gold badges364 silver badges416 bronze badges
answered Apr 30 ’16 at 11:24
JimJim8808 silver badges13 bronze badges
5
In case someone has tried all of these older answers, and is still running into problems like:
nnectionError:
SOCKSHTTPConnectionPool(host=’myhost’, port=80):
Max retries exceeded with url: /my/path
(Caused by NewConnectionError(‘< object at 0x106812bd0>:
Failed to establish a new connection:
[Errno 8] nodename nor servname provided, or not known’, ))
It may be because, by default, requests is configured to resolve DNS queries on the local side of the connection.
Try changing your proxy URL from socks5proxyhost:1234 to socks5hproxyhost:1234. Note the extra h (it stands for hostname resolution).
The PySocks package module default is to do remote resolution, and I’m not sure why requests made their integration this obscurely divergent, but here we are.
answered Apr 6 ’17 at 21:38
8
You need install pysocks, my version is 1. 0 and the code works for me:
import socket
import socks
ip=’localhost’ # change your proxy’s ip
port = 0000 # change your proxy’s port
tdefaultproxy(OXY_TYPE_SOCKS5, ip, port)
= cksocket
url = u”
print((url))
answered Oct 21 ’15 at 7:53
lqhcpsgbllqhcpsgbl3, 4743 gold badges19 silver badges30 bronze badges
2
As soon as python requests will be merged with SOCKS5 pull request it will do as simple as using proxies dictionary:
#proxy
# SOCKS5 proxy for HTTP/HTTPS
proxies = {
”: “socks5myproxy:9191″,
”: “socks5myproxy:9191″}
#headers
headers = {}
url=”
res = (url, headers=headers, proxies=proxies)
See SOCKS Proxy Support
Another options, in case that you cannot wait request to be ready, when you cannot use requesocks – like on GoogleAppEngine due to the lack of pwd built-in module, is to use PySocks that was mentioned above:
Grab the file from the repo and put a copy in your root folder;
Add import socks and import socket
At this point configure and bind the socket before using with urllib2 – in the following example:
import urllib2
t_default_proxy(CKS5, “”, port=9050)
res=urllib2. urlopen(url)()
answered Apr 5 ’16 at 10:41
loretoparisiloretoparisi13. 4k10 gold badges84 silver badges115 bronze badges
proxiesDict = {
”: “socks51. 2. 3. 4:1080″,
”: “socks51. 4:1080″}
# SOCKS4 proxy for HTTP/HTTPS
”: “socks41. 4:1080″,
”: “socks41. 4:1080″}
# HTTP proxy for HTTP/HTTPS
”: “1. 4:1080″,
”: “1. 4:1080”}
answered Aug 28 ’15 at 9:35
wcc526wcc5263, 2672 gold badges28 silver badges28 bronze badges
I installed pysocks and monkey patched create_connection in urllib3, like this:
tdefaultproxy(OXY_TYPE_SOCKS4, “127. 0. 1”, 1080)
def create_connection(address, timeout=socket. _GLOBAL_DEFAULT_TIMEOUT,
source_address=None, socket_options=None):
“””Connect to *address* and return the socket object.
Convenience function. Connect to *address* (a 2-tuple “(host,
port)“) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by:func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection.
An host of ” or port 0 tells the OS to use the default.
“””
host, port = address
if artswith(‘[‘):
host = (‘[]’)
err = None
for res in taddrinfo(host, port, 0, CK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = cksocket(af, socktype, proto)
# If provided, set socket level options before connecting.
# This is the only addition urllib3 makes to this function.
(sock, socket_options)
if timeout is not socket. _GLOBAL_DEFAULT_TIMEOUT:
ttimeout(timeout)
if source_address:
(source_address)
nnect(sa)
return sock
except as e:
err = e
if sock is not None:
()
if err is not None:
raise err
raise (“getaddrinfo returns an empty list”)
# monkeypatch
= create_connection
answered Feb 20 ’16 at 17:17
You can just run your script with _proxy environment variable.
Install socks support if it necessary.
pip install PySocks
pip install pysocks5
Setup environment variable
export _proxy=socks5
echo Your real IP
python -c ‘import requests;print((“))’
echo IP with socks-proxy
answered Jun 6 at 12:42
I could do this on Linux.
$ pip3 install –user ‘requests[socks]’
$ _proxy=socks5
> ‘import requests;print((“))’
answered Feb 20 ’20 at 17:48
Not the answer you’re looking for? Browse other questions tagged python proxy socks python-requests or ask your own question.
How to make python Requests work via socks proxy | Newbedev
Solution:The modern way:
pip install -U requests[socks]
then
import requests
resp = (”,
proxies=dict(=’socks5user:[email protected]:port’,
=’socks5user:[email protected]:port’))
As of requests version 2. 10. 0, released on 2016-04-29, requests supports SOCKS.
It requires PySocks, which can be installed with pip install pysocks.
Example usage:
proxies = {”: “socks5myproxy:9191″}
(”, proxies=proxies)
In case someone has tried all of these older answers, and is still running into problems like:
nnectionError:
SOCKSHTTPConnectionPool(host=’myhost’, port=80):
Max retries exceeded with url: /my/path
(Caused by NewConnectionError(‘< object at 0x106812bd0>:
Failed to establish a new connection:
[Errno 8] nodename nor servname provided, or not known’, ))
It may be because, by default, requests is configured to resolve DNS queries on the local side of the connection.
Try changing your proxy URL from socks5proxyhost:1234 to socks5hproxyhost:1234. Note the extra h (it stands for hostname resolution).
The PySocks package module default is to do remote resolution, and I’m not sure why requests made their integration this obscurely divergent, but here we are.
How to Use Socks Proxy with Request Module in Python
In one of our previous tutorial we learned about HTTP proxy. In this tutorial we will see another type of proxy called SOCKS. A proxy or proxy server is a computer that sits between you and the web server. It acts as a gateway between a local network and the internet. A proxy server works by intercepting connections between sender and receiver. All incoming data enters through one port and is forwarded to the rest of the network via another from traffic forwarding, proxy servers provide security by hiding the actual IP address of a, which stands for Socket Secure, is a network protocol that facilitates communication with servers through a firewall by routing network traffic to the actual server on behalf of a client. SOCKS is designed to route any type of traffic generated by any protocol or addition to basic HTTP proxies, Requests also supports proxies using the SOCKS protocol. This is an optional feature that requires that additional third-party libraries be installed before can get the dependencies for this feature from pip:pip install requests[socks]Using the scheme socks5 causes the DNS resolution to happen on the client, rather than on the proxy server. This is in line with curl, which uses the scheme to decide whether to do the DNS resolution on the client or proxy. If you want to resolve the domains on the proxy server, use socks5h as the to send requests through a SOCKS Proxy in Python 3 using RequestsThere are many websites dedicated to providing free proxies on the internet. One such site is. Go to the website and select a working proxy and put it in the below install requests[socks]
import requests
url = ”
proxies = {
“”:’socks5174. 76. 35. 15:36163′,
“”:’socks5174. 15:36163′}
response = (url, proxies=proxies)
Rotating Requests through a pool of Proxies in Python 3Rotating proxies is good habit to avoid getting blocked from website as most websites actively try to block ’ll gather a list of some active proxies from. To do that we will scrape this website. You can also use private proxies if you have access to we can see from below image the IP addresses are in a form of table and we need to get them from first column:import requests
from bs4 import BeautifulSoup as soup
response = (url)
bsobj = soup(ntent)
proxies= set()
for ip in ndAll(‘table’)[0]. findAll(‘tbody’)[0]. findAll(‘tr’):
cols = ndChildren(recursive = False)
cols = [() for element in cols]
#print(cols)
proxy = ‘:'([cols[0], cols[1]])
proxy = ‘socks4’+proxy
(proxy)
print(proxy)
Output:Now that we have the list of Proxy IP Addresses in a variable proxies, we’ll go ahead and rotate it using a Round Robin method and send get requests
from itertools import cycle
#If you are copy pasting proxy ips, put in the list below
#proxies = [‘121. 129. 127. 209:80’, ‘124. 41. 215. 238:45169’, ‘185. 93. 3. 123:8080’, ‘194. 182. 64. 67:3128’, ‘106. 0. 38. 174:8080’, ‘163. 172. 175. 210:3128′, ’13. 92. 196. 150:8080’]
proxy_pool = cycle(proxies)
for i in range(1, 10):
#Get a proxy from the pool
proxy = next(proxy_pool)
print(“Request #%d”%i)
try:
response = (url, proxies={“”: proxy, “”: proxy})
print(())
except:
#Most free proxies will often get connection errors. You will have retry the entire request using another proxy to work.
#We will just skip retries as its beyond the scope of this tutorial and we are only downloading a single url
print(“Skipping. Connnection error”)
Output:As you can see only 3 of our request were successful that is because most free proxies expire quickly. If you need to scrape large amount of data it is recommended to use paid proxies.