Welcome, guest | Sign In | My Account | Store | Cart

This recipe lets you take into account time spent in C modules when profiling your Python code. Normally the profiler only profiles Python code, so finding out how much time is spent accessing a database, running encryption code, sleeping and so on is difficult. Profilewrap makes it easy to profile C code as well as Python code, giving you a clearer picture of where your application is spending its time.

Profilewrap demonstrates how to create proxy objects at runtime that intercept calls between pre-existing pieces of code. It also demonstrates the use of the 'new' module to create new functions on the fly.

Python, 100 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
"""profilewrap.py:
Wraps C functions, objects and modules in dynamically-generated Python code
so you can profile them.  Here's an example using the rotor encryption module:

>>> import profilewrap, rotor, profile
>>> r = profilewrap.wrap( rotor.newrotor( 'key' ) )
>>> profile.run( "r.encrypt( 'Plaintext' )" )

This will produce output including something like this:

    1    0.003    0.003    0.003    0.003 PW_rotor.py:1(encrypt)

See the function _profileMe for examples of wrapping C functions, objects and
modules.  Run profilewrap.py to see the output from profiling _profileMe."""

import new, types

def _functionProxy( f, *args, **kwargs ):
   """The prototype for the dynamic Python code wrapping each C function."""
   return apply( f, args, kwargs )

class _ProfileWrapFunction:
   """A callable object that wraps each C function we want to profile."""
   def __init__( self, wrappedFunction, parentName="unnamed" ):
      # Build the code for a new wrapper function, based on _functionProxy.
      filename = "PW_%s.py" % parentName
      name = wrappedFunction.__name__
      c = _functionProxy.func_code
      newcode = new.code( c.co_argcount, c.co_nlocals, c.co_stacksize,
                          c.co_flags, c.co_code, c.co_consts, c.co_names,
                          c.co_varnames, filename, name, 1, c.co_lnotab )
      
      # Create a proxy function using the new code.
      self._wrapper = new.function( newcode, globals() )
      self._wrappedFunction = wrappedFunction
      
   def __call__( self, *args, **kwargs ):
      return apply( self._wrapper, (self._wrappedFunction,) + args, kwargs )


class _ProfileWrapObject:
   """A class that wraps an object or a module, and dynamically creates a
      _ProfileWrapFunction for each method.  Wrappers are cached for speed."""
   def __init__( self, wrappedObject ):
      self._wrappedObject = wrappedObject
      self._cache = {}
      
   def __getattr__( self, attrName ):
      # Look for a cached reference to the attribute and if it isn't there,
      # fetch it from the wrapped object.
      notThere = 'Not there'
      returnAttr = self._cache.get( attrName, notThere )
      if returnAttr is notThere:
         attr = getattr( self._wrappedObject, attrName, notThere )
         if attr is notThere:
            # The attribute is missing - let it raise an AttributeError.
            getattr( self._wrappedObject, attrName )
         
         # We only wrap C functions, which have the type BuiltinMethodType.
         elif isinstance( attr, types.BuiltinMethodType ):
            # Base the fictitious filename on the module name or class name.
            if isinstance( self._wrappedObject, types.ModuleType ):
               objectName = self._wrappedObject.__name__
            else:
               objectName = type( self._wrappedObject ).__name__
            returnAttr = _ProfileWrapFunction( attr, objectName )
            self._cache[ attrName ] = returnAttr
         
         # All non-C-function attributes get returned directly.
         else:
            returnAttr = attr
         
      return returnAttr


def wrap( wrappee ):
   """Wrap the given object, module or function in a Python wrapper."""
   if isinstance( wrappee, types.BuiltinFunctionType ):
      return _ProfileWrapFunction( wrappee )
   else:
      return _ProfileWrapObject( wrappee )

def _profileMe():
   # Wrap a built-in C function.
   wrappedEval = wrap( eval )
   print wrappedEval( '1+2*3' )
   
   # Replace a C module with its wrapped equivalent.
   import os
   os = wrap( os )
   print os.getcwd()

   # Wrap a C object.
   import rotor
   r = wrap( rotor.newrotor( 'key' ) )
   print repr( r.encrypt( 'Plaintext' ) )
   
if __name__ == '__main__':
   import profile
   profile.run( '_profileMe()' )

Here's a small piece of code using the 'rotor' encryption module:

import rotor, profile r = rotor.newrotor( 'key' ) profile.run( "r.encrypt( 'Plaintext' )" )

This won't produce any profiler output for the 'encrypt' method because it belongs to a C object. Profilewrap can wrap up the rotor object in dynamically-generated Python code which is accessible to the profiler, allowing you to do this:

import rotor, profile, profilewrap r = rotor.newrotor( 'key' ) r = profilewrap.wrap( r ) # << profilewrap in action, replacing 'r' with a profilable wrapper profile.run( "r.encrypt( 'Plaintext' )" )

You can now see an entry something like this in the profiler output:

1 0.003 0.003 0.003 0.003 PW_rotor.py:1(encrypt)

The filename PW_rotor.py is derived from the name of the object or module to which the method belongs - a 'rotor' object in this case. PW_rotor.py doesn't actually exist.

As well as objects, you can wrap individual functions [ eg. sleep=profilewrap.wrap(time.sleep) ] and whole modules [ eg. os=profilewrap.wrap(os) ] (Note that wrapping a module only wraps the functions that the modules exports - it doesn't automatically wrap objects created by those functions.) See '_profileMe' in profilewrap.py for examples.

2 comments

a 13 years, 11 months ago  # | flag

apply() is deprecated. apply( f, args, kwargs ) should be changed to apply( f, *args, **kwargs ). apply( self._wrapper, (self._wrappedFunction,) + args, kwargs ) should be changed to self._wrapper(self._wrappedFunction, *args, **kwargs ).

a 13 years, 11 months ago  # | flag

apply() is deprecated. apply( f, args, kwargs ) should be changed to f( args, *kwargs ). apply( self._wrapper, (self._wrappedFunction,) + args, kwargs ) should be changed to self._wrapper(self._wrappedFunction, args, *kwargs ).