Welcome, guest | Sign In | My Account | Store | Cart

Config - reads a configuration file.

This module parses a configuration file using a restricted Python syntax. The Python tokenizer/parser is used to read the file, unwanted expressions are removed from the parser output before the result is compiled and executed. The initialised configuration settings are returned in a dict.

Python, 541 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
# Config - reads a configuration file.
#
# This module parses a configuration file using a restricted Python syntax.
# The Python tokenizer/parser is used to read the file, unwanted expressions
# are removed from the parser output before the result is compiled and
# executed. The initialised configuration settings are returned in a dict.

import parser
import compiler
import symbol
import token
 
def _get_forbidden_symbols():
    """
    Returns a list of symbol codes representing statements that are *not*
    wanted in configuration files.
    """
    try:
        # Python 2.5:
        symlst = [symbol.break_stmt, symbol.classdef, symbol.continue_stmt,
                  symbol.decorator, symbol.decorators, symbol.eval_input,
                  symbol.except_clause, symbol.exec_stmt, symbol.flow_stmt,
                  symbol.for_stmt, symbol.fpdef, symbol.fplist, symbol.funcdef,
                  symbol.global_stmt, symbol.import_as_name, symbol.import_as_names,
                  symbol.import_from, symbol.import_name, symbol.import_stmt,
                  symbol.lambdef, symbol.old_lambdef, symbol.print_stmt,
                  symbol.raise_stmt, symbol.try_stmt, symbol.while_stmt,
                  symbol.with_stmt, symbol.with_var, symbol.yield_stmt,
                  symbol.yield_expr]
        
    except AttributeError:
        # Python 2.4:        
        symlst = [symbol.break_stmt, symbol.classdef, symbol.continue_stmt,
                  symbol.decorator, symbol.decorators, symbol.eval_input,
                  symbol.except_clause, symbol.exec_stmt, symbol.flow_stmt,
                  symbol.for_stmt, symbol.fpdef, symbol.fplist, symbol.funcdef,
                  symbol.global_stmt, symbol.import_as_name, symbol.import_as_names,
                  symbol.import_from, symbol.import_name, symbol.import_stmt,
                  symbol.lambdef, symbol.print_stmt, symbol.raise_stmt,
                  symbol.try_stmt, symbol.while_stmt]
    
    return symlst

def _parseconf(confstr):
    """
    Parse the configuration *confstr* string and remove anything else
    than the supported constructs, which are:

    Assignments, bool, dict list, string, float, bool, and, or, xor,
    arithmetics, string expressions and if..then..else.

    The *entire* statement containing the unsupported statement is removed
    from the parser; the effect is that the whole expression is ignored
    from the 'root' down.  

    The modified AST object is returned to the Python parser for evaluation. 
    """
    # Initialise the parse tree, convert to list format and get a list of
    # the symbol ID's for the unwanted statements. Might raise SyntaxError.
    
    ast = parser.suite(confstr)
    #ast = parser.expr(confstr)
    stmts = parser.ast2list(ast)
    rmsym = _get_forbidden_symbols()

    result = list()
    
    # copy 256: 'single_input', 257: 'file_input' or 258: 'eval_input'. The
    # parse tree must begin with one of these to compile back to an AST obj.

    result.append(stmts[0])

    # NOTE: This might be improved with reduce(...) builtin!? How can we get
    #       line number for better warnings?

    for i in range(1, len(stmts)):
        # censor the parse tree produced from parsing the configuration.
        if _check_ast(stmts[i], rmsym):
            result.append(stmts[i])
        else:
            pass
        
    return parser.sequence2ast(result)

def _check_ast(astlist, forbidden):
    """
    Recursively check a branch of the AST tree (in list form) for "forbidden"
    symbols. A token terminates the search.

    Returns False if any "forbidden symbols" are present, True otherwise. 
    """
    # check for token - tokens are always allowed.
    if astlist[0] in token.tok_name.keys():
        return True
    
    elif astlist[0] in forbidden:
        raise UserWarning('Statement containing '+symbol.sym_name[astlist[0]]
            +' not allowed, ignored!')
        return False
    
    else:
        return _check_ast(astlist[1], forbidden)
                  
def parse_config(confstr):
    """
    Use *eval(...)* to execute a filtered AST tree formed by parsing a
    configuration file, removing unwanted expressions (if any) and then
    compiling the filtered output to Python byte code. This approach
    allows the use of Python expressions and comments in config files,
    avoids the use of modules which is not particularily pretty (IMO).

    The following expressions (and combinations of) are allowed:

    Assignments, Arithmetic, Strings, Lists, if...then...else and
    Comments.

    Returns a dict containing the configuration values, if successful. 
    """
    dummy  = dict()
    result = dict()

    # Parse the python source code into a filtered AST-tree.
    cast = _parseconf(confstr)

    # Compile AST to bytecode, this also detects syntax errors.
    cobj = parser.compileast(cast)

    # Run the bytecode. The dicts *dummy* and *results* are placeholders
    # for the *globals* *locals* environments used by *eval(...)*. The
    # variables declared and initialised in the config file will end up
    # in *locals* together with the *__globals__* environment while the
    # *locals* will contain only the values created by *eval(...)*. This
    # is good, because we can protect *__globals__* and return only the
    # configuration values by doing this:
    
    eval(cobj, dummy, result)

    return result

# EOF


# <--------- split here ----------->


#!/usr/bin/env python
"""
Unit test for configuration handling. This is intended to be
run through the *tests* package, but should work stand-alone.
"""

import unittest
import sys, os
import tempfile
import string

# Adjust the import path to get the module we want to test.
# We are sitting in '<module>\\test' so one directory up
# will do the trick.

mod_path = os.path.normpath(os.getcwd()+'/..')
sys.path.append(mod_path)

import config

# import modules used internally by config.py.
import parser
import compiler
import symbol
import token
 

# Test: Assignments, bool, dict list, string, float, bool, and,
# or, xor,arithmetics, string expressions and if..then..else.
#
# The test code is mostly ripped from 'test_grammar.py', available
# from the Python source tree. 

test_backslash_1 = r"""
x = 1 \
+ 1
"""
test_backslash_2 = r"""
# Backslash does not means continuation in comments :\
x = 0
"""

test_integers_1 = r"""
a = 0xff
b = 0377
c = 2147483647
"""

test_long_ints_1 = r"""
x = 0L
x = 0l
x = 0xffffffffffffffffL
x = 0xffffffffffffffffl
x = 077777777777777777L
x = 077777777777777777l
x = 123456789012345678901234567890L
x = 123456789012345678901234567890l
"""

test_string_literals_1 = r"""
x = ''; y = ""
"""
test_string_literals_2 = r"""
x = '\''; y = "'"
"""
test_string_literals_3 = r"""
x = '"'; y = "\""
"""
test_string_literals_4 = r"""
x = "doesn't \"shrink\" does it"
y = 'doesn\'t "shrink" does it'
"""
test_string_literals_5 = r"""
x = "does \"shrink\" doesn't it"
y = 'does "shrink" doesn\'t it'
"""
test_string_literals_6 = r'''
x = r"""
The "quick"
brown fox
jumps over
the 'lazy' dog.
"""
y = '\nThe "quick"\nbrown fox\njumps over\nthe \'lazy\' dog.\n'
'''
test_string_literals_7 = r"""
y = '''
The "quick"
brown fox
jumps over
the 'lazy' dog.
'''
"""
test_string_literals_8 = r'''
y = "\n\
The \"quick\"\n\
brown fox\n\
jumps over\n\
the 'lazy' dog.\n\
"
'''
test_string_literals_9 = r"""
y = '\n\
The \"quick\"\n\
brown fox\n\
jumps over\n\
the \'lazy\' dog.\n\
'
"""

test_if_stmt_1 = r"""
if 1: pass
if 1: pass
else: pass
if 0: pass
elif 0: pass
if 0: pass
elif 0: pass
elif 0: pass
elif 0: pass
else: pass
"""

test_and_or_not_1 = r"""
if not 1: pass
if 1 and 1: pass
if 1 or 1: pass
if not not not 1: pass
if not 1 and 1 and 1: pass
if 1 and 1 or 1 and 1 and 1 or not 1 and 1: pass
"""

test_comparison_1 = r"""
if 1: pass
x = (1 == 1)
if 1 == 1: pass
if 1 != 1: pass
if 1 <> 1: pass
if 1 < 1: pass
if 1 > 1: pass
if 1 <= 1: pass
if 1 >= 1: pass
if 1 is 1: pass
if 1 is not 1: pass
if 1 in (): pass
if 1 not in (): pass
if 1 < 1 > 1 == 1 >= 1 <= 1 <> 1 != 1 in 1 not in 1 is 1 is not 1: pass
"""

test_binary_ops_1 = r"""
x = 1 & 1
x = 1 ^ 1
x = 1 | 1
"""

test_shift_ops_1 = r"""
x = 1 << 1
x = 1 >> 1
x = 1 << 1 >> 1
"""

test_additive_ops_1 = r"""
x = 1
x = 1 + 1
x = 1 - 1 - 1
x = 1 - 1 + 1 - 1 + 1
"""

test_multiplicative_ops_1 = r"""
x = 1 * 1
x = 1 / 1
x = 1 % 1
x = 1 / 1 * 1 % 1
"""

test_unary_ops_1 = r"""
x = +1
x = -1
x = ~1
x = ~1 ^ 1 & 1 | 1 & 1 ^ -1
x = -1*1/1 + 1*1 - ---1*1
"""

test_stmt_suite_1 = r"""
if 1: pass
if 1:
    pass
if 1:
#
#
#
    pass
pass
#
pass
#
"""

test_atoms_1 = r"""
x = (1)
x = (1 or 2 or 3)
x = (1 or 2 or 3, 2, 3)

x = []
x = [1]
x = [1 or 2 or 3]
x = [1 or 2 or 3, 2, 3]
x = []

x = {}
x = {'one': 1}
x = {'one': 1,}
x = {'one' or 'two': 1 or 2}
x = {'one': 1, 'two': 2}
x = {'one': 1, 'two': 2,}
x = {'one': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5, 'six': 6}

x = `x`
x = `1 or 2 or 3`
x = x
x = 'x'
x = 123
"""

class test_config_internals(unittest.TestCase):
    """
    Verify the internal functions in the config module. 
    """
    def test_get_forbidden_symbols(self):
        # verify that we can get the symbol ID's.
        res = config._get_forbidden_symbols()
        self.failUnless(len(res) > 0)
        
        # verify that the forbidden symbols are valid.
        for i in range(0, len(res)):
            self.failUnless(res[i] in symbol.sym_name.keys())

### BUG? Unittest flags expected exception as Error!
###        
##    def test_warning_on_banned_statement(self):
##        d = config._get_forbidden_symbols()
##        self.failUnlessRaises(UserWarning,
##            config._check_ast([d[0],[d[1]]], d))

    def test_parseconf(self):
        pass

    def test_check_ast(self):
        pass

class test_config(unittest.TestCase):
    """
    Verify the functionality of the configuration parser for
    a range of different data types and statements.
    """

    # Only use setUp() and tearDown() if necessary

    def setUp(self):
        pass
    
    def tearDown(self):
        pass
    
##    def _remove_rc_files(self, root):
##        # wipe out the temp files we created for the file load 
##        # tests.
##        #
##        for root, dirs, files in os.walk(root, topdown=False):
##            for name in files:
##                os.remove(os.path.join(root, name))
##            for name in dirs:
##                os.rmdir(os.path.join(root, name))
##
##    def _create_rc_files(self):
##        # create a set of rc-files containing test data
##        # and return the path to them.
##
##        rc_data_path = [
##            os.path.join(self.tmpdir, 'rc_data_0.conf'),
##            os.path.join(self.tmpdir, 'rc_data_1.conf'),
##            os.path.join(self.tmpdir, 'rc_data_2.conf')]
##        
##        rc_data = {
##            rc_data_path[0]: defaults.get_config_file_str(alpha),
##            rc_data_path[1]: defaults.get_config_file_str(beta),
##            rc_data_path[2]: defaults.get_config_file_str(gamma)}
##
##        for path in rc_data_path:
##            try:
##                fp = open(path, 'w')
##                fp.write(rc_data[path])
##            finally:
##                fp.close()
##
##        return rc_data_path

    def test_backslash(self):
        # Backslash means line continuation:
        res = config.parse_config(test_backslash_1)
        self.failUnless(res['x'] == 2)

        res = config.parse_config(test_backslash_2)
        self.failUnless(res['x'] == 0)

    def test_integers(self):
        # hex, octal and large positive ints.
        res = config.parse_config(string.lstrip(test_integers_1))
        self.failUnless(res['a'] == 255 and res['b'] == 255
                   and res['c'] == 017777777777)

    def test_long_ints(self):
        # test that the longint formats parses.
        res = config.parse_config(test_long_ints_1)
 
    def test_string_literals(self):
        # test some string definitions.
        res = config.parse_config(test_string_literals_1)
        self.failUnless(len(res['x']) == 0 and res['x'] == res['y'])

###BUG? Single quote ' seems to translate into "!!
###       
        res = config.parse_config(test_string_literals_2)
        self.failUnless(len(res['x']) == 1
                        and res['x'] == res['y'] and ord(res['x']) == 39)

        res = config.parse_config(test_string_literals_3)
        self.failUnless(len(res['x']) == 1
                        and res['x'] == res['y'] and ord(res['x']) == 34)
        
        res = config.parse_config(test_string_literals_4)
        self.failUnless(len(res['x']) == 24 and res['x'] == res['y'])

        res = config.parse_config(test_string_literals_5)
        self.failUnless(len(res['x']) == 24 and res['x'] == res['y'])

        res = config.parse_config(test_string_literals_6)
        self.failUnless(res['x'] == res['y'])

        res = config.parse_config(test_string_literals_6
                                  + test_string_literals_7)
        self.failUnless(res['x'] == res['y'])

        res = config.parse_config(test_string_literals_6
                                  + test_string_literals_8)
        self.failUnless(res['x'] == res['y'])

        res = config.parse_config(test_string_literals_6
                                  + test_string_literals_9)
        self.failUnless(res['x'] == res['y'])
        
### BUG? Unittest flags expected exception as Error!
###        
##    def test_syntax_error(self):
##        self.failUnlessRaises(SyntaxError,
##                              config.parse_config("a + 1 = b + 2"))
##        self.failUnlessRaises(SyntaxError,
##                              config.parse_config("x + 1 = 1"))
 
    def test_if_stmt(self):
        res = config.parse_config(test_if_stmt_1)

    def test_and_or_not(self):
        res = config.parse_config(test_and_or_not_1)

    def test_comparison(self):
        res = config.parse_config(test_comparison_1)

    def test_binary_ops(self):
        res = config.parse_config(test_binary_ops_1)

    def test_shift_ops(self):
        res = config.parse_config(test_shift_ops_1)

    def test_additive_ops(self):
        res = config.parse_config(test_additive_ops_1)

    def test_multiplicative_ops(self):
        res = config.parse_config(test_multiplicative_ops_1)

    def test_unary_ops(self):
        res = config.parse_config(test_unary_ops_1)

    def test_stmt_suite(self):
        res = config.parse_config(test_stmt_suite_1)

    def test_atoms(self):
        res = config.parse_config(test_atoms_1)

 
# run tests as default.
#
if __name__ == '__main__':
    unittest.main()

# EOF.

I needed a configuration file and did not want to write yet-another-parser for it and document the format. Instead of putting up with ConfigParser I decided to use the syntax and parser already available in Python, but restricted to a "safe" subset of the Python syntax: Assignments, bool, dict list, string, float, bool, and, or, xor, arithmetics, string expressions and if..then..else.

The "unsafe" Python statements are deleted from the code by editing the Python parse tree. The parse is one-way, one can only read config files. It might be possible to revert an AST tree to Pyton code, but I thought it would be messy.

The configuration is passed to the parser as a string either from a file or a string constant containing the default values.

Note: The source is actually in two parts: The configuration parser and the test script for it. The test script resides one directory "below" the code.

PS: The difference between parser.suite and parser.expr is not quite clear to me. As I understood the documentation, I should uses parser.expr but parser.suite is the one that works.

3 comments

David Moss 13 years, 4 months ago  # | flag

Have you thought about releasing this as a proper project rather than just a recipe? I'm personally loathe to use something with no prospects of bug tracking, improvements etc. The idea seems useful, but it needs some work.

The following would be problematic in a production setting :-

if __name__ == '__main__':
    cfg = """m = __import__('os'); m.unlink('remove_me.txt')"""
    #   Goodbye file ...
    parse_config(cfg)

A straight 'import os' is not allowed but access via the __import__ BIF still works (OUCH)!

Frithiof andreas Jensen (author) 13 years, 3 months ago  # | flag

PS: I think that one cannot do real security inside a module like this - the OS should do that. BUT - I think that the logic must be inverted so that only approved statements are permitted instead of the way it is now, where un-approved statments are disabled, creating holes.

Frithiof andreas Jensen (author) 13 years, 3 months ago  # | flag

I have thought about making up a module with distutil and all - I just want some critisim like yours, thankyou for that, before spending more time on something that is lame for obvious reasons (to everyone better than me). I will try the Python group and hear what they say too.