Python Http Library

Datacenter proxies

  • HTTP & SOCKS
  • unlimited bandwidth
  • Price starting from $0.08/IP
  • Locations: EU, America, Asia

Visit fineproxy.de

http.client — HTTP protocol client — Python 3.10.0 …

Source code: Lib//
This module defines classes which implement the client side of the HTTP and
HTTPS protocols. It is normally not used directly — the module
quest uses it to handle URLs that use HTTP and HTTPS.
See also
The Requests package
is recommended for a higher-level HTTP client interface.
Note
HTTPS support is only available if Python was compiled with SSL support
(through the ssl module).
The module provides the following classes:
class (host, port=None, [timeout, ]source_address=None, blocksize=8192)¶
An HTTPConnection instance represents one transaction with an HTTP
server. It should be instantiated passing it a host and optional port
number. If no port number is passed, the port is extracted from the host
string if it has the form host:port, else the default HTTP port (80) is
used. If the optional timeout parameter is given, blocking
operations (like connection attempts) will timeout after that many seconds
(if it is not given, the global default timeout setting is used).
The optional source_address parameter may be a tuple of a (host, port)
to use as the source address the HTTP connection is made from.
The optional blocksize parameter sets the buffer size in bytes for
sending a file-like message body.
For example, the following calls all create instances that connect to the server
at the same host and port:
>>> h1 = (”)
>>> h2 = (”)
>>> h3 = (”, 80)
>>> h4 = (”, 80, timeout=10)
Changed in version 3. 2: source_address was added.
Changed in version 3. 4: The strict parameter was removed. HTTP 0. 9-style “Simple Responses” are
not longer supported.
Changed in version 3. 7: blocksize parameter was added.
class (host, port=None, key_file=None, cert_file=None, [timeout, ]source_address=None, *, context=None, check_hostname=None, blocksize=8192)¶
A subclass of HTTPConnection that uses SSL for communication with
secure servers. Default port is 443. If context is specified, it
must be a LContext instance describing the various SSL
options.
Please read Security considerations for more information on best practices.
Changed in version 3. 2: source_address, context and check_hostname were added.
Changed in version 3. 2: This class now supports HTTPS virtual hosts if possible (that is,
if ssl. HAS_SNI is true).
no longer supported.
Changed in version 3. 4. 3: This class now performs all the necessary certificate and hostname checks
by default. To revert to the previous, unverified, behavior
ssl. _create_unverified_context() can be passed to the context
parameter.
Changed in version 3. 8: This class now enables TLS 1. 3
_handshake_auth for the default context or
when cert_file is passed with a custom context.
class (sock, debuglevel=0, method=None, url=None)¶
Class whose instances are returned upon successful connection. Not
instantiated directly by user.
Changed in version 3. 9 style “Simple Responses” are
This module provides the following function:
Parse the headers from a file pointer fp representing a HTTP
request/response. The file has to be a BufferedIOBase reader
(i. e. not text) and must provide a valid RFC 2822 style header.
This function returns an instance of
that holds the header fields, but no payload
(the same as
and).
After returning, the file pointer fp is ready to read the HTTP body.
parse_headers() does not parse the start-line of a HTTP message;
it only parses the Name: value lines. The file has to be ready to
read these field lines, so the first line should already be consumed
before calling the function.
The following exceptions are raised as appropriate:
exception ¶
The base class of the other exceptions in this module. It is a subclass of
Exception.
A subclass of HTTPException.
A subclass of HTTPException, raised if a port is given and is either
non-numeric or empty.
A subclass of ImproperConnectionState.
A subclass of HTTPException. Raised if a server responds with a HTTP
status code that we don’t understand.
A subclass of HTTPException. Raised if an excessively long line
is received in the HTTP protocol from the server.
A subclass of ConnectionResetError and BadStatusLine. Raised
by tresponse() when the attempt to read the response
results in no data read from the connection, indicating that the remote end
has closed the connection.
New in version 3. 5: Previously, BadStatusLine(”) was raised.
The constants defined in this module are:

The default port for the HTTP protocol (always 80).
The default port for the HTTPS protocol (always 443).
This dictionary maps the HTTP 1. 1 status codes to the W3C names.
Example: [] is ‘Not Found’.
See HTTP status codes for a list of HTTP status codes that are
available in this module as constants.
HTTPConnection Objects¶
HTTPConnection instances have the following methods:
quest(method, url, body=None, headers={}, *, encode_chunked=False)¶
This will send a request to the server using the HTTP request
method method and the selector url.
If body is specified, the specified data is sent after the headers are
finished. It may be a str, a bytes-like object, an
open file object, or an iterable of bytes. If body
is a string, it is encoded as ISO-8859-1, the default for HTTP. If it
is a bytes-like object, the bytes are sent as is. If it is a file
object, the contents of the file is sent; this file object should
support at least the read() method. If the file object is an
instance of io. TextIOBase, the data returned by the read()
method will be encoded as ISO-8859-1, otherwise the data returned by
read() is sent as is. If body is an iterable, the elements of the
iterable are sent as is until the iterable is exhausted.
The headers argument should be a mapping of extra HTTP headers to send
with the request.
If headers contains neither Content-Length nor Transfer-Encoding,
but there is a request body, one of those
header fields will be added automatically. If
body is None, the Content-Length header is set to 0 for
methods that expect a body (PUT, POST, and PATCH). If
body is a string or a bytes-like object that is not also a
file, the Content-Length header is
set to its length. Any other type of body (files
and iterables in general) will be chunk-encoded, and the
Transfer-Encoding header will automatically be set instead of
Content-Length.
The encode_chunked argument is only relevant if Transfer-Encoding is
specified in headers. If encode_chunked is False, the
HTTPConnection object assumes that all encoding is handled by the
calling code. If it is True, the body will be chunk-encoded.
Chunked transfer encoding has been added to the HTTP protocol
version 1. 1. Unless the HTTP server is known to handle HTTP 1. 1,
the caller must either specify the Content-Length, or must pass a
str or bytes-like object that is not also a file as the
body representation.
New in version 3. 2: body can now be an iterable.
Changed in version 3. 6: If neither Content-Length nor Transfer-Encoding are set in
headers, file and iterable body objects are now chunk-encoded.
The encode_chunked argument was added.
No attempt is made to determine the Content-Length for file
objects.
tresponse()¶
Should be called after a request is sent to get the response from the server.
Returns an HTTPResponse instance.
Note that you must have read the whole response before you can send a new
request to the server.
Changed in version 3. 5: If a ConnectionError or subclass is raised, the
HTTPConnection object will be ready to reconnect when
a new request is sent.
t_debuglevel(level)¶
Set the debugging level. The default debug level is 0, meaning no
debugging output is printed. Any value greater than 0 will cause all
currently defined debug output to be printed to stdout. The debuglevel
is passed to any new HTTPResponse objects that are created.
New in version 3. 1.
t_tunnel(host, port=None, headers=None)¶
Set the host and the port for HTTP Connect Tunnelling. This allows running
the connection through a proxy server.
The host and port arguments specify the endpoint of the tunneled connection
(i. the address included in the CONNECT request, not the address of the
proxy server).
The headers argument should be a mapping of extra HTTP headers to send with
the CONNECT request.
For example, to tunnel through a HTTPS proxy server running locally on port
8080, we would pass the address of the proxy to the HTTPSConnection
constructor, and the address of the host that we eventually want to reach to
the set_tunnel() method:
>>> import
>>> conn = (“localhost”, 8080)
>>> t_tunnel(“)
>>> quest(“HEAD”, “/”)
New in version 3. 2.
nnect()¶
Connect to the server specified when the object was created. By default,
this is called automatically when making a request if the client does not
already have a connection.
()¶
Close the connection to the server.
HTTPConnection. blocksize¶
Buffer size in bytes for sending a file-like message body.
New in version 3. 7.
As an alternative to using the request() method described above, you can
also send your request step by step, by using the four functions below.
HTTPConnection. putrequest(method, url, skip_host=False, skip_accept_encoding=False)¶
This should be the first call after the connection to the server has been
made. It sends a line to the server consisting of the method string,
the url string, and the HTTP version (HTTP/1. 1). To disable automatic
sending of Host: or Accept-Encoding: headers (for example to accept
additional content encodings), specify skip_host or skip_accept_encoding
with non-False values.
Send an RFC 822-style header to the server. It sends a line to the server
consisting of the header, a colon and a space, and the first argument. If more
arguments are given, continuation lines are sent, each consisting of a tab and
an argument.
Send a blank line to the server, signalling the end of the headers. The
optional message_body argument can be used to pass a message body
associated with the request.
If encode_chunked is True, the result of each iteration of
message_body will be chunk-encoded as specified in RFC 7230,
Section 3. 3. How the data is encoded is dependent on the type of
message_body. If message_body implements the buffer interface the encoding will result in a single chunk.
If message_body is a, each iteration
of message_body will result in a chunk. If message_body is a
file object, each call to () will result in a chunk.
The method automatically signals the end of the chunk-encoded data
immediately after message_body.
Due to the chunked encoding specification, empty chunks
yielded by an iterator body will be ignored by the chunk-encoder.
This is to avoid premature termination of the read of the request by
the target server due to malformed encoding.
New in version 3. 6: Chunked encoding support. The encode_chunked parameter was
added.
(data)¶
Send data to the server. This should be used directly only after the
endheaders() method has been called and before getresponse() is
called.
HTTPResponse Objects¶
An HTTPResponse instance wraps the HTTP response from the
server. It provides access to the request headers and the entity
body. The response is an iterable object and can be used in a with
statement.
Changed in version 3. 5: The io. BufferedIOBase interface is now implemented and
all of its reader operations are supported.
([amt])¶
Reads and returns the response body, or up to the next amt bytes.
adinto(b)¶
Reads up to the next len(b) bytes of the response body into the buffer b.
Returns the number of bytes read.
New in version 3. 3.
Return the value of the header name, or default if there is no header
matching name. If there is more than one header with the name name,
return all of the values joined by ‘, ‘. If ‘default’ is any iterable other
than a single string, its elements are similarly returned joined by commas.
Return a list of (header, value) tuples.
Return the fileno of the underlying socket.
A instance containing the response
headers. is a subclass of
ssage.
rsion¶
HTTP protocol version used by server. 10 for HTTP/1. 0, 11 for HTTP/1. 1.
URL of the resource retrieved, commonly used to determine if a redirect was followed.
Headers of the response in the form of an Message instance.
Status code returned by server.
Reason phrase returned by server.
buglevel¶
A debugging hook. If debuglevel is greater than zero, messages
will be printed to stdout as the response is read and parsed.
Is True if the stream is closed.
Deprecated since version 3. 9: Deprecated in favor of url.
Deprecated since version 3. 9: Deprecated in favor of headers.
tstatus()¶
Deprecated since version 3. 9: Deprecated in favor of status.
Examples¶
Here is an example session that uses the GET method:
>>> conn = (“)
>>> quest(“GET”, “/”)
>>> r1 = tresponse()
>>> print(, )
200 OK
>>> data1 = () # This will return entire content.
>>> # The following example demonstrates reading data in chunks.
>>> while chunk:= (200):… print(repr(chunk))
b’\n>> # Example of an invalid request
>>> conn = (“”)
>>> r2 = tresponse()
404 Not Found
>>> data2 = ()
>>> ()
Here is an example session that uses the HEAD method. Note that the
HEAD method never returns any data.
>>> res = tresponse()
>>> data = ()
>>> print(len(data))
0
>>> data == b”
True
Here is an example session that shows how to POST requests:
>>> import,
>>> params = ({‘@number’: 12524, ‘@type’: ‘issue’, ‘@action’: ‘show’})
>>> headers = {“Content-type”: “application/x-www-form-urlencoded”,… “Accept”: “text/plain”}
>>> quest(“POST”, “”, params, headers)
>>> response = tresponse()
302 Found
>>> data
b’Redirecting to

Requests: HTTP for Humans™ — Requests 2.26.0 …

Release v2. 26. 0. (Installation)
Requests is an elegant and simple HTTP library for Python, built for human beings.
Behold, the power of Requests:
>>> r = (”, auth=(‘user’, ‘pass’))
>>> atus_code
200
>>> r. headers[‘content-type’]
‘application/json; charset=utf8’
>>> r. encoding
‘utf-8’
>>>
‘{“type”:”User”… ‘
>>> ()
{‘private_gists’: 419, ‘total_private_repos’: 77,… }
See similar code, sans Requests.
Requests allows you to send HTTP/1. 1 requests extremely easily.
There’s no need to manually add query strings to your
URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling
are 100% automatic, thanks to urllib3.
Beloved Features¶
Requests is ready for today’s web.
Keep-Alive & Connection Pooling
International Domains and URLs
Sessions with Cookie Persistence
Browser-style SSL Verification
Automatic Content Decoding
Basic/Digest Authentication
Elegant Key/Value Cookies
Automatic Decompression
Unicode Response Bodies
HTTP(S) Proxy Support
Multipart File Uploads
Streaming Downloads
Connection Timeouts
Chunked Requests
Support
Requests officially supports Python 2. 7 & 3. 6+, and runs great on PyPy.
The User Guide¶
This part of the documentation, which is mostly prose, begins with some
background information about Requests, then focuses on step-by-step
instructions for getting the most out of Requests.
Installation of Requests
$ python -m pip install requests
Get the Source Code
Quickstart
Make a Request
Passing Parameters In URLs
Response Content
Binary Response Content
JSON Response Content
Raw Response Content
Custom Headers
More complicated POST requests
POST a Multipart-Encoded File
Response Status Codes
Response Headers
Cookies
Redirection and History
Timeouts
Errors and Exceptions
Advanced Usage
Session Objects
Request and Response Objects
Prepared Requests
SSL Cert Verification
Client Side Certificates
CA Certificates
Body Content Workflow
Keep-Alive
Streaming Uploads
Chunk-Encoded Requests
POST Multiple Multipart-Encoded Files
Event Hooks
Custom Authentication
Streaming Requests
Proxies
Compliance
HTTP Verbs
Custom Verbs
Link Headers
Transport Adapters
Blocking Or Non-Blocking?
Header Ordering
Authentication
Basic Authentication
Digest Authentication
OAuth 1 Authentication
OAuth 2 and OpenID Connect Authentication
Other Authentication
New Forms of Authentication
The API Documentation / Guide¶
If you are looking for information on a specific function, class, or method,
this part of the documentation is for you.
Developer Interface
Main Interface
Exceptions
Request Sessions
Lower-Level Classes
Lower-Lower-Level Classes
Encodings
Status Code Lookup
Migrating to 1. x
Migrating to 2. x
The Contributor Guide¶
If you want to contribute to the project, this part of the documentation is for
you.
Contributor’s Guide
Be Cordial
Get Early Feedback
Contribution Suitability
Code Contributions
Steps for Submitting Code
Code Review
New Contributors
Kenneth Reitz’s Code Style™
Documentation Contributions
Bug Reports
Feature Requests
Authors
Keepers of the Crystals
Previous Keepers of Crystals
Patches and Suggestions
There are no more guides. You are now guideless.
Good luck.
The best Python HTTP clients for 2021 - ScrapingBee

The best Python HTTP clients for 2021 – ScrapingBee


Updated:
28 June, 2021
13 min read
Ian is a freelance developer with a passion for simple solutions. He has written code to power surveys, studio pipelines and holds a PhD in distributed computing.
There are a huge number of HTTP clients available for Python – a quick search for Python HTTP Clients on Github returns over 1700 results(! ) How do you make sense of all of them and find one which is right for your particular use case?
Do you have a single machine at your disposal, or a collection of them? Do you want to keep things simple or is raw performance more of a concern? A web application needing to make the odd request to a micro-service api is going to have quite different requirements to a script constantly scraping data. Additionally, there’s the concern wether the library you choose will still be around 6 months down the line.
In this article we’re going to cover five of the best HTTP clients currently available for Python and detail why each of them might be one for you to consider.
Introduction
For all the examples here, I’ll be making GET requests to the Star Wars API (), which returns data about the people, planets and data from the Star Wars Universe. You can see an example of a JSON response from it below:
{
“name”: “Death Star”,
“model”: “DS-1 Orbital Battle Station”,
“manufacturer”: “Imperial Department of Military Research, Sienar Fleet Systems”,
“cost_in_credits”: “1000000000000”,… }
// Now I know which manufacturer I won’t be asking to make my own Death Star.
The POST request examples here are to which is a developer testing tool responding with the content of the request, you could also use if you prefer. We’ll be sending the following JSON POST data about Obi Wan:
“name”: “Obi-Wan Kenobi”,
“height”: “182”,
“mass”: “77”,
“hair_color”: “auburn, white”,
“skin_color”: “fair”,
“eye_color”: “blue-gray”,
“birth_year”: “57BBY”,
“gender”: “male”}
The Basics
If you’re familiar with Pythons standard library, you’re probably already aware of the confusing history of urllib and urllib2 modules within it. urllib2 (the original module) was split into separate modules in Python 3, quest and
For comparison purposes with the packages in the rest of this article, let’s first take a look at how we’d make a request using nothing but the standard library.
All our examples that follow use Python 3
import json
import quest
response = quest. urlopen(”)
text = ()
print(((‘utf-8’)))
Note how we’ve had to use the json module to convert this into json, as quest returns a byte literal.
Our POST would look like this:
from urllib import request, parse
data = {“name”: “Obi-Wan Kenobi”,… }
encoded_data = (data)()
req = quest(”, data=encoded_data)
d_header(‘Content-Type’, ‘application/json’)
response = request. urlopen(req)
We’ve also had to encode the data we want to send and set the header content type which we’d need to update if we were submitting form data for example.
You might be feeling this is clunky – “All I wanted was to get some data! ” Well, this is how many other developers felt too given with a number of HTTP clients available as additional packages. In the rest of the article we’ll take a look at 5 of the best choices available.
urllib3
urllib3 is a powerful, user-friendly HTTP client for Python. Much of the Python ecosystem already uses urllib3 and you should too. urllib3 brings many critical features that are missing from the Python standard librarie
The urllib3 package is rather confusingly not in the standard library, but a separate HTTP client package which builds upon urllib. It provides missing features such as connection pooling, TLS verification and thread safety. This ultimately results in better performance for applications making many calls like web scraping, as they will reuse connections to hosts rather than creating new ones.
It’s actually a dependency of the HTTP client requests listed later in this article and gets over 150M downloads a month. In order to make a request using urllib3, we’d make a call with it like the following:
import urllib3
= urllib3. PoolManager()
r = quest(‘GET’, ”)
As with the standard library, we’ve had to convert this to JSON ourselves as urllib3 leaves us to do things manually.
For any post requests, we’d also need to manually encode query parameters or JSON like so:
encoded_data = (data)(‘utf-8’)
r = quest(
‘POST’,
”,
body=encoded_data,
headers={‘Content-Type’: ‘application/json’})
The Poolmanager object in each example handles the connection pooling and thread safety with the request requiring supplying a HTTP verb as a string argument. This is an extra step that gives urllib3 many of its additional features. Pools will be cached so that subsequent requests to the same host will use the same connection instance. This means in order to make many requests to the same host, we might want to up the maximum size of the number of HTTPConnections.
urllib3 also offers complex retry behavior. This is a really important consideration – we don’t want our connection to timeout due to a random one-off overloaded server and then just give up. We’d like to try multiple times before we consider the data unavailable. You can read up on how to use these features within the urllib3 documentation if you’re interested.
The downside of using urllib3’s connection pooling is that it makes it difficult to work with cookies as it isn’t a stateful client. We have to manually set these as a header value on the request rather than having direct support by urllib3, or use something like the okies module to manage them for us. For example:
headers={‘Cookie’: ‘foo=bar; hello=world’}
Given that so many other libraries depend on urllib3, it’s likely it will exist for some time to come.
Requests
Requests is an elegant and simple HTTP library for Python, built for human beings.
The Requests package is highly favored within the Python community, garnering over 110M downloads a month according to PePy. It’s also recommended as a “higher level HTTP client interface” on the main quest documentation. Working with requests incredibly simple and as such the majority of developers in the Python community use it as their HTTP client of choice. It’s maintained by the Python Software Foundation with over 45k stars on Github and a dependency of many other python libraries such as gRPC and pandas.
Let’s review how we’d make our requests with Requests(! ):
import requests
r = (”)
print(())
Similarly, posting data is also made simple – we just need to change our get method call to a post:
r = (”, json=data)
Here you can see why requests is so popular – its design is just so elegant! The example here is the most concise requiring the least code of all the examples given so far. Requests incorporates HTTP verbs as methods (GET, POST) and we’ve even been able to convert straight to JSON without having to write our own decode method. As a developer this means it’s dead simple to work with and understand, with only two method calls necessary to get the data we want from our API.
Within our POST, we’ve also not had to bother with encoding our data dictionary or worry about setting correct content type in the request headers. Request does all that for us. Thanks Requests!
It’s also easy to modify our POST call to submit form data instead by simply replacing our ‘json’ argument with ‘data’.
Another example of its simplicity is the way we can set cookies which are just an additional argument on the post method. For example:
r = (”, data=data, cookies={‘foo’: ‘bar’, ‘hello’: ‘world’}))
Requests also offers a whole host of other advanced features like sessions, request hooks and custom retry strategies. Sessions allow for statefulness with cookies being persisted across each request from a session instance, something urllib3 didn’t provide. An example taken from the request documentation:
s = ssion()
(”)
print()
# ‘{“cookies”: {“sessioncookie”: “123456789”}}’
Additionally, hooks allow you to register common behavior you want to execute after each call. You may be familiar with this concept if you use git, which allows you to the same. You can check out all the advanced features within the requests documentation.
Given all requests advanced features means its a solid choice for a variety of applications.
AIOHTTP
Asynchronous HTTP Client/Server for asyncio and Python.
AIOHTTP is package containing both a client and server framework, meaning it might be well suited to an API which also makes requests elsewhere. It has 11k stars on Github and a number of third party libraries build upon it. Making a GET request with it is as follows:
import aio
import asyncio
async def main():
async with ientSession() as session:
async with (”) as response:
print(await ())
loop = t_event_loop()
n_until_complete(main())
and our POST:
async with (”, json=data) as response:
You can see that the ientSession() object uses similar syntax to requests – but the overall code is much longer than previous examples and now we have method calls using async, await along with an additional module import for asyncio.
The AIOHTTP documentation gives a good overview of why all this extra code is necessary compared to say requests. It will take some time to understand the asynchronous programming concepts if you’re not familiar with them, but what it ultimately means is it’s possible to make a number of requests at the same time without waiting for each to return a response one after another. For situations where we only make a single request this might not be a concern, but if we need to make tens or even thousands of requests, all the time the CPU is waiting for a response could be better spent doing something else (like making another request! ). We don’t want to be paying for CPU cycles when we’re just waiting around. As an example, let’s take a look at some code looking up data for the first 50 starships from the Star Wars API.
import time
async def get_starship(ship_id: int):
async with (f’ship_id}/’) as response:
tasks = []
for ship_id in range(1, 50):
(get_starship(ship_id))
await (*tasks)
(main())
This consistently took under 2s to run on my machine, whilst requesting the same data using a session with requests it takes just over 4s. So we’re able to speed up the time it takes to retrieve our data if we can deal with the additional complexity it introduces to our code.
AIOHTTP offers thorough documentation along with a host of advanced features like sessions, cookies, pools, dns caching and client tracing. Where it falls down however is its lack of support for complex retry behavior which is only available via third party modules.
GRequests
GRequests brings Gevent – a ‘coroutine -based Python networking library’ to requests to allow requests to be made asynchronously. It’s an older library, first released in 2012 which doesn’t use Pythons standard asyncio module. Individual requests can be made as we would do with requests, but we can also leverage the Gevent module to make a number of requests like so:
import grequests
reqs = []
for ship_id in range(0, 50):
((f’ship_id}/’))
for r in (reqs):
GRequests documentation is sparse and even goes as far to recommend other libraries over it on its Github page. At just 165 lines of code, it doesn’t offer any advanced functionality over Requests itself. Over it’s 9 years it’s had a total of 6 releases, so is probably only really worth considering if you find async programming particularly confusing.
HTTPX
HTTPX is the newest package on this list (first released in 2015) and at the time of writing is still in beta, with a v1 expected sometime in 2021. It offers a “broadly requests-compatible API”, is the only example here to offer HTTP2 support and also offers async APIs.
Using HTTPX is very similar to requests:
import x
and for our POST:
We’ve simply changed the name of our module and again not needed to manage any JSON conversion. You’ll also notice even though it has async apis, we can still write concise synchronous code with it. We can also create asynchronous versions of our examples, by using the yncClient and using requests style verb syntax against that. This allows us to get all the starship details we could ever desire in next to no time from our API. We update our previous get_starship method from the aio example to use x:
async with yncClient() as client:
r = await (f’ship_id}/’)
print(())…
For requests which take some time to return a response, this again means our client does not have to wait around. It’s definitely worth considering if you have a large number of requests you need to make simultaneously and want to save on CPU cycles. If you also are looking to refactor scripts based on requests to something asynchronous, then HTTPX would seem to be a good replacement.
Comparing Client Results
The benchmarks we’ve taken aren’t exhaustive and serve only as a guide. Each of our examples use Pythons time module to instrument a request call, execute it 100 times within a loop and return an average execution time. Where possible, we reuse clients and sessions for each package. For our asynchronous requests we measure the total execution time for 100 calls made asynchronously and return the average for each request. For the purposes of comparison and to negate server response time within the benchmarks, both GET and POST requests are made to a local instance. Download statistics for our comparison are taken from PePy which uses public PyPI download information.
Name
Downloads / mo
Github Stars
Sync GET
Sync POST
Async GET
Async POST
urllib
N/A
5. 79ms
6. 12ms
154M
2. 7k
4. 13ms
4. 39ms
115M
45. 5k
6. 94ms
7. 41ms
Grequests
0. 37M
3. 8k
7. 11ms
7. 66ms
4. 53ms
4. 95ms
25M
11. 3k
6. 07ms
6. 61ms
3. 58ms
3. 92ms
4M
7k
5. 43ms
5. 72ms
4. 01ms
4. 34ms
We can see that in terms of pure performance for individual requests, urllib3 is the winner – with POSTs taking less time than the other libraries. The difference in times of our GET requests is very minimal at just under 3ms between all 5 of the libraries. It’s interesting to note that although urllib3 may have less Github stars (sometimes taken as a indication of popularity), it’s actually downloaded the most (remember what I said about it being a dependency of Requests? ). Requests is clearly the library with the most community and offers a raft of advanced features we’ve not benchmarked in these simple tests.
In terms of async clients, AIOHTTP comes out on top, with the time per request for both GET and POSTs being the least. It also sports the most downloads and stars, but bear in mind this may well be because it also offers web framework behavior too. Due to the lack of activity on the GRequests project and it’s own advice, you probably shouldn’t consider it unless you have very specific needs.
Conclusion
We have seen throughout this article that Requests has inspired the design of many of the libraries shown. It is incredibly popular within the Python community, with it being the default choice for most developers. With the additional features it offers like sessions and simple retry behavior, you should likely be looking at it if you have simple needs or want to maintain simple code.
For those developers with more complex requirements and wanting to make more requests – AIOHTTP would currently be the best choice. Out of all our tests it performed the best asynchronously, it has the largest number of downloads/stars and currently offers a stable release. It is however complex with no retry support out of the box so maybe worth looking to HTTPX when a v1 release is available or if it being in beta is not a concern.
Whatever your client needs, there’s a Python package out there for you!

Frequently Asked Questions about python http library

What is a HTTP library Python?

The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.

How do you make an HTTP connection in Python?

Python – HTTP Clientimport requests r = requests. get(‘https://httpbin.org/’) print(r. text)[:200]import requests s = requests. Session() s. get(‘http://httpbin.org/cookies/set/sessioncookie/31251425’) r = s. get(‘http://httpbin.org/cookies’) print(r. text)requests. get(‘http://github.com’, timeout=10.001)

What is HTTP Python?

http is a package that collects several modules for working with the HyperText Transfer Protocol: http. client is a low-level HTTP protocol client; for high-level URL opening use urllib. request. … server contains basic HTTP server classes based on socketserver.

Leave a Reply

Your email address will not be published.