Python Pickle: examples and reference

Last updated:
Table of Contents

WIP Alert This is a work in progress. Current information is correct but more content may be added in the future.

Python 3 is used in all examples, unless explicitly noted

Pickle dict

Use a .p or .pkl extension to follow convention

import pickle

colors = { "john": "yellow", "mary": "red" }

pickle.dump(colors, open("colors.p", "wb"))

Unpickle dict

Pickles can be hacked so only read files you can trust

import pickle

# read back the file written by the method above
colors = pickle.load(open("colors.p","rb"))

Pickle Scikit-learn model

Just apply the method above after you have called fit() (or fit_transform() or partial_fit()) on your model:

import pickle
from sklearn.linear_model import LinearRegression

clf = LinearRegression()
clf.fit(<train_features>, <train_target>)

# using mode "wb" is important
pickle.dump(clf, open("/path/to/model.pkl", "wb"))

ModuleNotFoundError: No module named XYZ

TODO

PicklingError: Can't pickle function lambda

lambda functions can't be pickled because they all have the same name.

Just define a function using def instead

AttributeError: Can't get attribute 'MyClass' on module 'main'

Code in the root scope of classes can't be unpickled by default.

While you get no error if you pickle it, you can't normally unpickle it1

  • in one file:

    import pickle
    
    # pickle class
    class MyClass:
        attr1 = "some information"
    
    pickle.dump(MyClass,open("/tmp/kls.pkl","wb"))
    
  • in another file:

    import pickle
    
    unpickled_class = pickle.load(open("/tmp/kls.pkl","rb"))
    # >>> AttributeError                            Traceback (most recent call last)
    # <ipython-input-3-12b3f1c86c3a> in <module>
    #----> 1 unpickled_kls = pickle.load(open("/tmp/kls.pkl","rb"))
    
    #AttributeError: Can't get attribute 'MyClass' on <module '__main__'>
    

Heads-up: Use wb file mode

Pickle is a binary data format so be sure to read and write files using the binary flags ("rb" and "wb", respectively).


1: There are workarounds: pickling class instances

Dialogue & Discussion