I've written a small class to handle a gzip pipe that won't read the whole source file at once, but will deliver small chunks of data on demand.
Python, 81 lines
This is useful when writing a mono-thread server based on sockets. It improves performance a lot when dealing with very large files.
This class uses an internal buffer.
When the "read" method is called, it feeds the internal zip object that writes back to the buffer. When the buffer is large enough (larger than the requested size) the read method returns back the proper amount of data.
I would like to do the same job for tar files, but the tar module doesn't seem to provide a way to add just a chunk of data ; Only a whole file at once.
I'd be glad if someone could help.