Subprocess with async I/O pipes class (Python)

2009-05-17T02:02:04-07:00

Python recipe 576759 by Mike Kazantsev (asynchronous, ipc, non_blocking_i_o, pipe, subprocess). Revision 2.

Just stumbled upon the need to move data chunks between subprocesses in a non-linear way with some logic in-between, so tee(1) and fifo(7)'s weren't too good option. Inspired by 440554, but rewritten from scratch to remove unnecessary delays due to sleep(3) calls and suboptimal try/sleep-based polling.

Threaded communication with subprocess (Python)

2013-03-13T06:06:16-07:00

Python recipe 578488 by Jan Müller (async, ipc, messaging, multithreading, pipe, queue, subprocess). Revision 3.

This recipe shows how to domesticate another executable as a service in a subprocess.

Collection Pipeline in Python (Python)

2016-03-16T14:45:02-07:00

Python recipe 580625 by Steven D'Aprano (bash, filter, map, pipe, pipline).

A powerful functional programming technique is the use of pipelines of functions. If you have used shell scripting languages like bash, you will have used this technique. For instance, to count the number of files or directories, you might say: ls | wc -l. The output of the first command is piped into the input of the second, and the result returned.

We can set up a similar pipeline using Lisp-like Map, Filter and Reduce special functions. Unlike the standard Python map, filter and reduce, these are designed to operate in a pipeline, using the same | syntax used by bash and other shell scripting languages:

>>> data = range(1000)
>>> data | Filter(lambda n: 20 < n < 30) | Map(float) | List
[21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]

The standard Python functional tools is much less attractive, as you have to write the functions in the opposite order to how they are applied to the data. This makes it harder to follow the logic of the expression.

>>> list(map(float, filter(lambda n: 20 < n < 30, data)))
[21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]

We can also end the pipeline with a call to Reduce to collate the sequence into a single value. Here we take a string, extract all the digits, convert to ints, and multiply:

>>> from operator import mul
>>> "abcd12345xyz" | Filter(str.isdigit) | Map(int) | Reduce(mul)
120

Pyliner - Script to run arbitrary Python code on the command line (Python)

2011-07-29T13:22:54-07:00

Python recipe 577075 by Drew Gulino (awk, oneliner, perl, pipe, python, sed, sysadmin).

A Python script that runs arbitrary Python scripts in an input loop. This allows one-liner Python scripts similarly to how Perl runs them (-l,-a,-n,-F, BEGIN, END)

It provides additional syntax that allows multiline Python scripts to be run on a single line

CallPipe: Call the methods of an object in a remote process (Python)

2008-09-17T17:33:36-07:00

Python recipe 576509 by david decotigny (call, multiprocessing, pipe, processing, remote). Revision 6.

I am process A and I own object Obj. Now I decide to fork(): process B is born.

How do I do to make process B call the methods of A's object Obj (and not the methods of its own copy of Obj...) ?

This is what this module does: you create the "CallPipe" for Obj prior to fork()ing, and then process B can call any method of A's object Obj through it.

Python-controlled Unix pipeline to generate PDF (Python)

2016-01-07T18:02:52-08:00

Python recipe 579146 by Vasudev Ram (linux, pdf, pdf_generation, pipe, pipelining, python, python2, unix).

This recipe shows how to create a Unix pipeline that generates PDF output, under the control of a Python program. It is tested on Linux. It uses nl, a standard Linux command that adds line numbers to its input, and selpg, a custom Linux command-line utility, that selects only specified pages from its input, together in a pipeline (nl | selpg). The Python program sets up and starts that pipeline running, and then reads input from it and generates PDF output.

Asynchronous pipe communication using asyncore (Python)

2013-10-29T16:48:22-07:00

Python recipe 576967 by Glenn Eychaner (async, asyncore, i_o, pipe). Revision 9.

Extends file_dispatcher to provide extra functionality for reading from and writing to pipes. Uses the observer pattern (recipe 576962) to provide notification of new data and closed pipes.

Align text filter (Python)

2010-03-09T07:34:43-08:00

Python recipe 577093 by Denis Barmenkov (align, pipe, text, wrap). Revision 4.

Python 2.3 module textwrap can justify text in three modes: left/right/center. Sometimes 'align' mode become more useful.

commanding via a duplex pipe stream. (C)

2009-06-07T01:41:19-07:00

C recipe 576773 by J Y (command, ipc, pipe). Revision 2.

commanding via a duplex pipe stream.

rdd (mostly broken but shows how to do a few things in ruby) (Ruby)

2014-07-12T16:58:06-07:00

Ruby recipe 578034 by Mike 'Fuzzy' Partin (dd, ftp, io, net_ftp, pipe, pipelining, popen, subprocess). Revision 3.

Meant to be a slightly more "advanced" dd utility. Supporting FTP/File/STDIN as input streams, and File/STDOUT/PIPE as output targets, and sporting a progress display (very rudimentary atm), add lets you combine network, and file or pipe processing in a single command. But kind of ended up a mess, see the Python version which is pretty clean.

Functional D plus Python pipeline to generate PDF (Text)

2016-09-22T15:32:17-07:00

Text recipe 580701 by Vasudev Ram (pdfwriter, pdf_generation, pipe, pipeline, xtopdf).

This recipe is a command pipeline. The first component of the pipeline is a D language program that makes use of simple functional programming and template / generic programming features of D, to transform some input into the desired output. Both input and output are text. The D program writes the output to standard output, which is then read by a Python program that reads that as input via standard input, and converts it to PDF.

Most viewed recipes tagged "pipe"