Design pattern that is highly reusable. Simple handlers implement one specific task from a complex set of tasks to be performed on an object. Such handlers can then be layered in a stack, in different combinations, together achieving complex processing of an object. New handlers are easy to implement and add.
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | class Handler (object):
"""The base class for all the handlers"""
def __init__(self):
self.data = {}
self.nextHandler = None
def __add__(self, newHandler):
"""Used to append handlers to each other"""
if not isinstance(newHandler, Handler):
raise TypeError('Handler.__add__() expects Handler')
if self.nextHandler:
self.nextHandler + newHandler
else:
self.nextHandler = newHandler
while newHandler:
newHandler.data = self.data
newHandler = newHandler.nextHandler
return self
def useHook(self, fileName):
"""Wrapper around the hook method"""
if self.nextHandler:
if not self.nextHandler.useHook(fileName):
return False
else:
self.data.clear( )
return self.hook(fileName)
def hook(self, fileName):
"""Default hook method to be overridden in subclasses"""
return True
# Subclasses of Handler
import os, time
class Filter (Handler):
def hook(self, fileName):
if fileName[-3:] == '.py':
return True
else:
return False
class Processor (Handler):
def hook(self, fileName):
modtime = os.path.getmtime(fileName)
if (time.time()- modtime) < 24*60*60:
self.data['state']='changed'
self.data['modtime']=time.ctime(modtime)
else:
self.data['state']='unchanged'
return True
class Logger (Handler):
def hook(self, fileName):
print fileName,
for key in self.data.keys():
print '%s=%s' % (key, self.data[key]),
print
return True
# Script that shows the use of a handler stack
if '__main__'==__name__:
import sys
a=Logger( )
b=Processor( )
c=Filter( )
d=a+b+c
for dirPath, dirNames, fileNames in os.walk(sys.argv[1]):
for fileName in fileNames:
d.useHook(os.path.join(dirPath, fileName))
|
A good way to divide a set of operations or tasks that can be performed on an object into smaller, simpler handlers. Each handler implements only one task and several handlers can be reused in different configurations in order to implement a complex functionality. New handlers are easy to implement because they only override one or several hook methods from a base class, and the base class takes care of generic functionality.
Reusability is the big reason for using this pattern. For instance, a file filter can be reused with many other handlers that perform a task on files. And a set of small, specific filters can be layered together into more complex filters.
The mechanism chosen to append handlers to each other is to override the __add__ operator. This allows great flexibility in how handlers can be layered in the stack. For example,<pre> hStack = hInst1 + hInst2 + hInst3</pre> creates hStack from three instances of Handler subclasses. The same result would be achieved by:<pre> hInst1 + hInst2 + hInst3 hStack = hInst1</pre> Even more, the same result is achieved by:<pre> hStack = hInst1 hStack + hInst2 hStack + hInst3</pre> or by:<pre> hInst2 + hInst3 hStack = hInst1 + hInst2</pre> or even by: <pre> hStack = hInst1 + hInst2 hInst2 + hInst3</pre> The attribute 'data' in class Handler is a dictionary object that is shared by all the handlers in the stack. The __add__ operator ensures that. Data can be passed from a handler to handlers above it in the stack by writing entries in self.data.
A hook method can return a value of 'False' and thus act as a filter by blocking the handlers above it from processing the object. If it returns a value of 'True', a handler also has the ability to pass additional data to the handlers above it in the stack. Although the wrapper methods (useHook in this example) are invoked top-down, the hook methods are effectively invoked bottom-up and data can be passed only in that direction.
Note to the editors: this code snippet is the basic idea behind an open-source project that I am working on and that I "own" (see http://pyfmf.sourceforge.net). If the recipe is approved, I would appreciate it if the project would also be mentioned.
Great Contribution. Great code. I picked up a few new Python tricks, thanks! I hope your fmf project goes well.
Derrick
RE: Great Contribution. Thanks, Derrick. From your mentioning my project I figured that this time the comment is really addressed to me (for recipe #302422). But these comments are still getting posted also to another recipe (#302086). I sent a comment to Support and let's hope that they will fix it. I also hope that Derrick's comment will not get lost with the fix. Hear that, Support people? ;-)
Question on #302422 Handler Stack. Looks like a powerful use of overloading, but a single, perhaps obvious, question: all handlers in a stack have as their "data" member an alias to the same dictionary, right? Would be a useful clarification for some of us. Thanks in advance.
Re: Question on #302422 Handler Stack. You're right about the 'data' dictionary and you're right that it deserves an explanation, so I have changed the 'Discussion' to include that. Thanks for your comments.
While we are on the topic of changes, I should mention that I have made a few changes to the code since the initial submission. The more important ones are that I got rid of 'nextHandler' and 'data' as class attributes and I initialize them only as instance attributes. I decided it's better to avoid the class attributes altogether and initializing 'data' as a class attribute was actually a bug. I also added the raise of an exception in the __add__ operator if the argument is not a Handler instance. Such a wrong use of the __add__ operator was raising an exception anyway, but that exception was kind of confusing for debugging.
Thanks, and a few more thoughts... First, Dan, thanks. Second, a couple small things have popped into mind:
1) The class looks much better now that data is an instance variable...as you mention on your fmf project site, in some ways this shares a lot in common with a "chain of responsibility" pattern, but with a shared dictionary, no "buck stops here rule" and the possibility of additional handler-specific data this "handler stack" is nicely suited for broader situations.
2) A small, somewhat quibbling thought, for C++ users this would seem like an appropriate place to use an iostreams style bit shift operator overload... perhaps less confusing to leave the code as is, but unlike the addition operator, the bitshift operator makes it visually clearer (for c++ or shell script users, anyway) which element is where in the hierarchy of handlers. Obviously you use __add__ in your fmf project, but having __lshift__ = __add__ in the recipe might be a possibility if you aren't reserving the operator.
3) Depending on how confident you are in your users' capabilities, you might want to throw in an exception check to make sure that a handler never becomes its own parent. The occasional sanity check never hurts...
4) Again, this may be more trouble than it's worth--or run against the grain of your idea--but let's say that I want to create a complex set of handlers that, say, logs to a logfile (handler "l") and fills in class data (handler "c") based on file input (read in by handler "f"). Currently, I could stack handlers so that l = l + c + f and get the desired results. But what if I wanted to log all commands in the file but eliminate unsupported inputs by means of an intermediate (handler "p", proxy) that failed on encountering unsupported file data chunks. It would be nice to be able to do l + f and then c + p + f with the result that I create a complex, branched handler stack sharing a dictionary and instance of handler f's class between two filter chains. However, the current setup causes f to take c + p 's data dictionary instead of l's so that I would be forced to create a seperate instance of f's class for the second stack. If the size of the data dictionary grows large (i.e. reading in large data sets from nested file formats), keeping multiple copies or running multiple pass through input data could quickly start chewing up lots of memory/time. It might be helpful if the Handler and the nextHandler took on a merged value if their dictionaries differed rather than Handler dictating the contents of nextHandler's dictionary outright. Alternately, you could create a way to initialize new handler instances so that they used a preexisting stack's data. This way we could initialize handler c with its data dictionary set to be the same as handler l's and thereby avoid needless duplication of data in memory, though multiple passes through data might still be necess
(comment continued...)
(...continued from previous comment)
ary. (Obviously you could force this by simply setting data equal to the desired dictionary, but if you decide to obscure access to the data dictionary at a later point, it would be easier to have an approved way to do this now). These adjustments would allow users to build up some fairly elaborate "filter graph" style setups very elegantly.
Just some thoughts, LJ
Re: a few more thoughts... All great comments. The suggestions you're making are more relevant for my fmf project, I think the recipe is long and complex enough as it is. BTW, thanks for taking a look also at my project and this gives me a clue that I should add a contact info on the web site. Everyone, please use that contact for comments on my project if they are beyond the scope of this recipe.
Good point about the '__lshift__' vs. the '__add__' operators. The direction of the flow is not obvious with '+'. I'm not going to change it yet though, let's see what other people suggest. I agree that '__lshift__' would be more suggestive than '__add__' (at least to some people), but maybe someone will come up with something even better. Anyway, I would keep only one of them, it is confusing to have 2 operators for the same thing.
The elaborate 'filter graph' is something that I will probably have to address in the future, in my project. For now, I have a configuration mechanism that needs work even with a simple stack like this. Very good point about the need for more flexibility in setting the common 'data' dictionary and therefore the need for a method to set it outside of the '+' operator. Thanks, LJ.
+ is confusing and unnecessary. You have changed the meaning of the + operator here. The accepted meaning is "add these two items together without modifying either one, and return the result." The way you are using it means "modify B by adding A". As there is already an accepted operator for that (+=), I strongly suggest you implement __iadd__ (for A += B) instead of __add__.
Re: + is confusing and unnecessary. I am beginning to see even more that using the '+' operator may be confusing to some people. LJ's suggestion to use '__lshift__' is becoming more tempting, but an argument similar to yours can be made also against that operator.
BTW, I agree with you that this use of '+' is changing its sense and that may be a source of confusion. However, I would not replace it with '__iadd__' either because that would eliminate the creation of an entire stack in one line, like 'a+b+c+d+e'. And if I would go that way, I would rather just use a function, like a.append(b) or a.setNextHandler(b).
I guess that a compromise would be to change '__add__' to create a new Handler instead of changing the existing one in-place. That would eliminate some of the usages in the discussion, but that would be preferred if it also eliminates some confusion. The issue then is how to create a copy of a Handler. A copy constructor should be useful anyway and such a complete solution should definitely be considered in a real case implementation, but I think it would be too much for this recipe.
I'll still leave it as an open issue for now.
Side note: This discussion has given me the idea of making the stack iterable and adding an iterator for it (or making it its own iterator), but this would also be too much for the recipe.
alternative to + operator. I am adding an alternative to the + (__add__) operator because that seems to be an element of confusion:
This alternative also has many usages:
and so on.
I will not change the code in the recipe yet, but I am offering both alternatives for now. I will eventually change the code based on feedback and especially if the editors request it.
Re: alternative to + operator. Correction on usages (3rd line: Handler.makeStack(a,b), not Handler(a,b)):
For what it's worth... My preference would be to provide __iadd__ and chain methods. __iadd__ would be the same as the __add__ defined above. chain would look like:
This would provide the following possibilities:
or (getting a little carried away):
That seems to me to be a good compromise, and is less confusing (to me, at least).
Design note: For efficiency, I would also eliminate the duplicate type check introduced in chain(). I would extract all of __iadd__ except the type check into a separate method (maybe __iadd_no_typecheck), and have both __iadd__ and chain call the new routine.
pipes and filters. I have finally discovered a pattern that is well described and matches the architecture described in this recipe. It's the Pipes and Filters pattern described in "Pattern-Oriented Software Architecture, Volume 1" by Frank Buschmann and others. It was published in 1996 so I guess they were first ;-).