caching (Caching Module)

Author:

• Dirk Alders <sudo-dirk@mount-mockery.de>

Description:

This Module supports functions and classes for caching e.g. properties of other instances.

Submodules:

• caching.property_cache_json
• caching.property_cache_pickle

Unittest:

See also the unittest documentation.

class caching.property_cache_json(source_instance, cache_filename, load_all_on_init=False, callback_on_data_storage=None)

Class to cache properties, which take longer on initialising than reading a file in json format. See also parent property_cache_pickle

Parameters:

• source_instance (instance) – The source instance holding the data
• cache_filename (str) – File name, where the properties are stored as cache
• load_all_on_init – Optionally init behaviour control parameter. True will load all available properties from source on init, False not.

Warning

• This class uses json. You should only use keys of type string!
• Unicode types are transfered to strings

Note

source_instance needs to have at least the following methods: uid(), keys(), data_version(), get()

• uid(): returns the unique id of the source.
• keys(): returns a list of all available keys.
• data_version(): returns a version number of the current data (it should be increased, if the get method of the source instance returns improved values or the data structure had been changed).
• get(key, default): returns the property for a key. If key does not exists, default will be returned.

Reasons for updating the complete data set:

• UID of source_instance has changed (in comparison to the cached value).
• data_version is increased

Example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import sys
import time
sys.path.append('../..')

import caching
import report


report.stdoutLoggingConfigure(log_name_lvl=[('root', 'DEBUG'), ])


class test_slow_data(object):
    _ONE = '1'
    _TWO = '2'
    _THREE = '_property_cache_data_version_'
    _FOUR = '_property_cache_uid_'
    _FIVE = '__property_cache_uid_'
    KEYS = [_ONE, _TWO, _THREE, _FOUR, _FIVE]
    VERS = 0.1

    def data_version(self):
        return self.VERS

    def one(self):
        return self.get(self._ONE)

    def two(self):
        return self.get(self._TWO)

    def three(self):
        return self.get(self._THREE)

    def four(self):
        return self.get(self._FOUR)

    def five(self):
        return self.get(self._FIVE)

    def get(self, key, default=None):
        def print_n_sleep(k):
            sys.stdout.write('slow get executed for %s\n' % k)
            time.sleep(3)
        if key == self._ONE:
            print_n_sleep(key)
            return 'one'
        if key == self._TWO:
            print_n_sleep(key)
            return 'two'
        if key == self._THREE:
            print_n_sleep(key)
            return 'three'
        if key == self._FOUR:
            print_n_sleep(key)
            return 'four'
        if key == self._FIVE:
            print_n_sleep(key)
            return 'five'
        return default

    def keys(self):
        return self.KEYS

    def uid(self):
        return None


class tsd_cache_json(test_slow_data):
    def __init__(self, *args, **kwargs):
        test_slow_data.__init__(self, *args, **kwargs)
        self._cached_data = caching.property_cache_json(test_slow_data(*args, **kwargs), 'cache.json', load_all_on_init=False)

    def two(self):
        return test_slow_data.get(self, self._TWO)

    def get(self, key, default=None):
        return self._cached_data.get(key, default)


data = tsd_cache_json()
print('Testing property_cache (json):\\n--------------------------------')
print(data.one())
print(data.two())
print(data.three())
print(data.four())
print(data.five())

Will result on the first execution to the following output (with a long execution time):

Testing property_cache (json):\n--------------------------------
2023-10-26 00:00:39,938: root.caching - DEBUG - JsonCache: Cache file does not exists (yet).
2023-10-26 00:00:39,938: root.caching - INFO - JsonCache: cache-file stored (cache.json)
slow get executed for 1
2023-10-26 00:00:42,941: root.caching - DEBUG - JsonCache: Loading property for '1' from source instance ('one')
2023-10-26 00:00:42,942: root.caching - INFO - JsonCache: cache-file stored (cache.json)
one
slow get executed for 2
two
slow get executed for _property_cache_data_version_
2023-10-26 00:00:48,946: root.caching - DEBUG - JsonCache: Loading property for '_property_cache_data_version_' from source instance ('three')
2023-10-26 00:00:48,946: root.caching - INFO - JsonCache: cache-file stored (cache.json)
three
slow get executed for _property_cache_uid_
2023-10-26 00:00:51,950: root.caching - DEBUG - JsonCache: Loading property for '_property_cache_uid_' from source instance ('four')
2023-10-26 00:00:51,950: root.caching - INFO - JsonCache: cache-file stored (cache.json)
four
slow get executed for __property_cache_uid_
2023-10-26 00:00:54,954: root.caching - DEBUG - JsonCache: Loading property for '__property_cache_uid_' from source instance ('five')
2023-10-26 00:00:54,954: root.caching - INFO - JsonCache: cache-file stored (cache.json)
five

With every following execution (slow for getting “two” which is not cached - see implementation):

Testing property_cache (json):\n--------------------------------
2023-10-26 00:00:55,004: root.caching - INFO - JsonCache: Loading properties from cache (cache.json)
2023-10-26 00:00:55,004: root.caching - DEBUG - JsonCache: Providing property for '1' from cache ('one')
one
slow get executed for 2
two
2023-10-26 00:00:58,007: root.caching - DEBUG - JsonCache: Providing property for '_property_cache_data_version_' from cache ('three')
three
2023-10-26 00:00:58,007: root.caching - DEBUG - JsonCache: Providing property for '_property_cache_uid_' from cache ('four')
four
2023-10-26 00:00:58,007: root.caching - DEBUG - JsonCache: Providing property for '__property_cache_uid_' from cache ('five')
five

class caching.property_cache_pickle(source_instance, cache_filename, load_all_on_init=False, callback_on_data_storage=None)

Class to cache properties, which take longer on initialising than reading a file in pickle format.

Parameters:

• source_instance (instance) – The source instance holding the data
• cache_filename (str) – File name, where the properties are stored as cache
• load_all_on_init – Optionally init behaviour control parameter. True will load all available properties from source on init, False not.

Note

source_instance needs to have at least the following methods: uid(), keys(), data_version(), get()

• uid(): returns the unique id of the source.
• keys(): returns a list of all available keys.
• data_version(): returns a version number of the current data (it should be increased, if the get method of the source instance returns improved values or the data structure had been changed).
• get(key, default): returns the property for a key. If key does not exists, default will be returned.

Reasons for updating the complete data set:

• UID of source_instance has changed (in comparison to the cached value).
• data_version is increased

Example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import sys
import time
sys.path.append('../..')

import caching
import report


report.stdoutLoggingConfigure(log_name_lvl=[('root', 'DEBUG'), ])


class test_slow_data(object):
    _ONE = '1'
    _TWO = '2'
    _THREE = '_property_cache_data_version_'
    _FOUR = '_property_cache_uid_'
    _FIVE = '__property_cache_uid_'
    KEYS = [_ONE, _TWO, _THREE, _FOUR, _FIVE]
    VERS = 0.1

    def data_version(self):
        return self.VERS

    def one(self):
        return self.get(self._ONE)

    def two(self):
        return self.get(self._TWO)

    def three(self):
        return self.get(self._THREE)

    def four(self):
        return self.get(self._FOUR)

    def five(self):
        return self.get(self._FIVE)

    def get(self, key, default=None):
        def print_n_sleep(k):
            sys.stdout.write('slow get executed for %s\n' % k)
            time.sleep(3)
        if key == self._ONE:
            print_n_sleep(key)
            return 'one'
        if key == self._TWO:
            print_n_sleep(key)
            return 'two'
        if key == self._THREE:
            print_n_sleep(key)
            return 'three'
        if key == self._FOUR:
            print_n_sleep(key)
            return 'four'
        if key == self._FIVE:
            print_n_sleep(key)
            return 'five'
        return default

    def keys(self):
        return self.KEYS

    def uid(self):
        return None


class tsd_cache_pickle(test_slow_data):
    def __init__(self, *args, **kwargs):
        test_slow_data.__init__(self, *args, **kwargs)
        self._cached_data = caching.property_cache_pickle(test_slow_data(*args, **kwargs), 'cache.pickle', load_all_on_init=False)

    def two(self):
        return test_slow_data.get(self, self._TWO)

    def get(self, key, default=None):
        return self._cached_data.get(key, default)


data = tsd_cache_pickle()
print('Testing property_cache (pickle):\\n--------------------------------')
print(data.one())
print(data.two())
print(data.three())
print(data.four())
print(data.five())

Will result on the first execution to the following output (with a long execution time):

Testing property_cache (pickle):\n--------------------------------
2023-10-26 00:00:21,374: root.caching - DEBUG - PickCache: Cache file does not exists (yet).
2023-10-26 00:00:21,375: root.caching - INFO - PickCache: cache-file stored (cache.pickle)
slow get executed for 1
2023-10-26 00:00:24,378: root.caching - DEBUG - PickCache: Loading property for '1' from source instance ('one')
2023-10-26 00:00:24,378: root.caching - INFO - PickCache: cache-file stored (cache.pickle)
one
slow get executed for 2
two
slow get executed for _property_cache_data_version_
2023-10-26 00:00:30,385: root.caching - DEBUG - PickCache: Loading property for '_property_cache_data_version_' from source instance ('three')
2023-10-26 00:00:30,385: root.caching - INFO - PickCache: cache-file stored (cache.pickle)
three
slow get executed for _property_cache_uid_
2023-10-26 00:00:33,389: root.caching - DEBUG - PickCache: Loading property for '_property_cache_uid_' from source instance ('four')
2023-10-26 00:00:33,389: root.caching - INFO - PickCache: cache-file stored (cache.pickle)
four
slow get executed for __property_cache_uid_
2023-10-26 00:00:36,392: root.caching - DEBUG - PickCache: Loading property for '__property_cache_uid_' from source instance ('five')
2023-10-26 00:00:36,828: root.caching - INFO - PickCache: cache-file stored (cache.pickle)
five

With every following execution (slow for getting “two” which is not cached - see implementation):

Testing property_cache (pickle):\n--------------------------------
2023-10-26 00:00:36,879: root.caching - INFO - PickCache: Loading properties from cache (cache.pickle)
2023-10-26 00:00:36,880: root.caching - DEBUG - PickCache: Providing property for '1' from cache ('one')
one
slow get executed for 2
two
2023-10-26 00:00:39,883: root.caching - DEBUG - PickCache: Providing property for '_property_cache_data_version_' from cache ('three')
three
2023-10-26 00:00:39,883: root.caching - DEBUG - PickCache: Providing property for '_property_cache_uid_' from cache ('four')
four
2023-10-26 00:00:39,883: root.caching - DEBUG - PickCache: Providing property for '__property_cache_uid_' from cache ('five')
five

get(key, default=None)

Method to get the cached property. If key does not exists in cache, the property will be loaded from source_instance and stored in cache (file).

Parameters:

• key – key for value to get.
• default – value to be returned, if key does not exists.

Returns:

value for a given key or default value.

keys()

Method to get the available keys (from source_instance).