Caching Function Results in Python: Examples using Joblib and FunctoolsLast updated:
All examples use Python 3.5+
Decorators in Python are functions that take other functions as parameters 1
See all examples here: caching-stuff-notebook
This is better suited for functions that use small objects as parameters (primitive values like strings, integers, etc) and return small objects.
import functools import time # maxsize=4 means that you will only # cache up to 4 different results @functools.lru_cache(maxsize=4) def myfunc(x): # using time.sleep to simulate an operation # that takes a long time time.sleep(2) # then return the result return x # this takes 2 seconds myfunc(1) # >>> 1 # the results for myfunc(1) are now cached: # this returns instantaneously myfunc(1) # >>> 1 myfunc(2) # >>> 2 myfunc(3) # >>> 3 myfunc(4) # >>> 4 # At this point, the value for myfunc(5) is added to # the cache, and the value for myfunc(1) is removed. myfunc(5) # >>> 5
Since our cache only caches up to 4 results, calling
myfunc(1) to be removed from the cache. The results for
myfunc(1) are removed because it's the least-recently used (LRU) result.
@memory.cache decorator, found in the
This is better suited for functions that take large objects as parameters and return large objects too.
The actual results are cached as pickle files, in directory
joblib/, under the location you passed as
from joblib import Memory import time memory = Memory(location="/path/to/cache/directory",verbose=0) # same example as above @memory.cache def myfunc(x): # using time.sleep to simulate an operation # that takes a long time time.sleep(2) # then return the result return x
1: These are also called higher-order functions in functional programming languages.
- you can pass other options to the Memory class, such as the maximum size to be used for caching, etc.