Pickling Error When Multithreading - Design Or Code Issue?

I'm writing a tool in pygtk which needs to deal with recursively parsing large directories, and adding resulting matched files to a list. This process obviously causes the user int

Solution 1:

I don't know well GTK, but I think your problem is more about pickling than about multiprocessing.

The __getstate__ and __setstate__ methods of the pickle module lets you customize the pickling process for any object.

Here is a trivial example which show how it works :

from pickle import dumps, loads


classNotPickable(object):
    def__init__(self, x):
        self.attr = x

ffile = open('/tmp/filesarenotpickable', 'r+w')    
o = NotPickable(ffile)
dumps(o)
# =>  TypeError: can't pickle file objectsclassPickable(NotPickable):
    attr = open('/tmp/a_file_on_an_other_system', 'r+w')

    def__getstate__(self):
        return self.attr.read()

    def__setstate__(self, state):
        self.attr.write(state)

o = Pickable(ffile)                                            
dumps(o)
# OUT: 'ccopy_reg\n_reconstructor\np0\n(c__main__\nPickable\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'                        

o2 = loads(dumps(o))                                           
o2.attr
# OUT: <open file '/tmp/a_file_on_an_other_system', mode 'r+w' at 0x18ad4b0>

Of course, it remains the responsibility of the developer to represent and properly restore the state of objects.