Welcome, guest | Sign In | My Account | Store | Cart

Line-oriented processing in Python from command line (like AWK) (Python recipe) by Artur Siekielski
ActiveState Code (http://code.activestate.com/recipes/577656/)

A very simple but powerful shell script which enables writing ad-hoc Python scripts for processing line-oriented input. It executes the following code template:

$INIT
for line in sys.stdin:
    $LOOP
$END

where $INIT, $LOOP and $END code blocks are given from command line. If only one argument is given, then $INIT and $END are empty. If two arguments are given, $END is empty.

Examples (script is saved as 'pyk' in the $PATH):

  • "wc -l" replacement: $ cat file | pyk 'c=0' 'c+=1' 'print c'
  • grep replacement: $ cat file | pyk 'import re' 'if re.search("\d+", line): print line'
  • adding all numbers: $ seq 1 10 | pyk 's=0' 's+=int(line)' 'print s'
  • prepending lines with it's length: $ cat file | pyk 'print len(line), line'
  • longest file name: $ ls -1 | pyk 'longest=""' 'if len(line) > len(longest): longest=line' 'print longest'
  • number of unique words in a document: $ pyk 'words=[]' 'words.extend(line.split())' 'print "All words: {}, unique: {}".format(len(words), len(set(words))'
Python, 34 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#!/bin/bash

if [ "$#" -eq "0" ]; then
    echo "Args: <INIT> <LOOP> <END>. <LOOP> code can use 'line' variable"
    echo "If -d is given as a first arg, code to be executed is printed"
    exit 1
fi

if [ "$1" = "-d" ]; then
    DEBUG=1
    shift
fi

[ "$#" -eq "1" ] && LOOP="$1"
[ "$#" -eq "2" ] && INIT="$1" && LOOP="$2"
[ "$#" -eq "3" ] && INIT="$1" && LOOP="$2" && END="$3"

CODE="
import sys

$INIT

for line in sys.stdin:
    line = line.strip()
    $LOOP

$END
"

if [ "$DEBUG" = "1" ]; then
    echo $CODE
fi

exec python -c "$CODE"
Tags: awk, bash, python, shell

1 comment

Oivvio Polite 12 years, 12 months ago  # | flag

This simple recipe will really change my ability to use python for command line stuff. Thanks you very much!