Chunked Transfer Encoding in Python Web Applications

Chunked Transfer Encoding is a method introduced in HTTP 1.1 for sending data as a series of successive chunks. The Content-Length header is not set, therefore nor the sender nor the receiver need to know the size of the entire request beforehand. This allows the sender to start transmitting without buffering the entire request in-memory.

To tell the server that the request is being sent in chunks, the client sets the Transfer-Encoding: chunked header. The size of the chunk is then sent in the body alongside the chunk divided by a line separator, so the body of a request would look something like this.


In the above example, the size of each chunk is 7 bytes, with one character accounting for one byte.

 Reading Chunked Requests

As chunked transfer is not part of the WSGI specification, dealing with such requests is quite tricky. For example: Flask reports the body of the request as empty because of the missing (or zero) Content-Length header.

The pieces of code below are based on Flask, but they should be fairly easy to convert to whatever framework is being used. All that needs to change is how to access the environ variables.

Also, the code is used to create a generator, as that was my use case.

 Apache and mod_wsgi

I haven’t been able to get Apache with mod_wsgi to correctly read the entire body while in daemon mode, and this piece of code only works in embedded mode.

In the site configuration file, add WSGIChunkedRequest On.

def chunked_reader():
    stream = flask.request.environ["wsgi.input"]
        while True:



uWSGI worked generally better. Chunked input needs to be enabled either through the command line argument --http-chunked-input or through the configuration file by a similarly named option.

When running a python application under uWSGI, the server imports a special module called uwsgi which includes server-specific functions. In this case we’re interested in the chunked_read() function. There is also a non-blocking function called chunked_read_nb().

import uwsgi

def chunked_reader()
    while True:
        chunk = uwsgi.chunked_read()
        if len(chunk) > 0:
            yield chunk



Now read this

Hats? Global Game Jam 2015

Had such a blast at the Global Game Jam 2015 @ Northeastern University as part of Team Boulder (alongside Matt Long, Nicole Amidon, Walton Lee, Nathan Low, Sarah Zaidan, Rui Jiang). We developed a 2D side-scrolling platformer in Unity... Continue →