rez.package_cache#

class rez.package_cache.PackageCache#

Bases: object

Package cache.

A package cache is responsible for storing copies of variant payloads into a location that would typically be on local disk. The intent is to avoid fetching a package’s files over shared storage at runtime.

A package cache is used like so:

A rez-env is performed;
The context is resolved;
For each variant in the context, we check to see if it’s present in the current package cache;
If it is, the variant’s root is remapped to this location.

A package cache is not a package repository. It just stores copies of variant payloads - no package definitions are stored.

Payloads are stored into the following structure:

/<cache_dir>/foo/1.0.0/af8d/a/<payload>
                           /a.json

Here, ‘af8d’ is the first 4 chars of the SHA1 hash of the variant’s ‘handle’, which is a dict of fields that uniquely identify the variant. To avoid hash collisions, the variant is then stored under a subdir that is incrementally named (‘a’, ‘b’, …, ‘aa’, ‘ab’, …). The ‘a.json’ file is used to find the correct variant within the hash subdir. The intent is to keep cached paths short, and avoid having to search too many variant.json files to find the matching variant.

VARIANT_NOT_FOUND = 0#: Variant was not found

VARIANT_FOUND = 1#: Variant was found

VARIANT_CREATED = 2#: Variant was created

VARIANT_COPYING = 3#: Variant payload is still being copied to this cache

VARIANT_COPY_STALLED = 4#: Variant payload copy has stalled

VARIANT_PENDING = 5#: Variant is pending caching

VARIANT_REMOVED = 6#: Variant was deleted

__init__(path)#

Create a package cache.

Parameters:: path (str) – Path on disk, must exist.

get_cached_root(variant)#

Get location of variant payload copy.

Parameters:: variant (Variant) – Variant to search for.
Returns:: Cached variant root path, or None if not found.
Return type:: str

add_variant(variant, force=False)#

Copy a variant’s payload into the cache.

The following steps are taken to ensure muti-thread/proc safety, and to guarantee that a partially-copied variant payload is never able to be used:

The hash dir (eg ‘/<cache_dir>/foo/1.0.0/af8d’) is created;
A file lock mutex (‘/<cache_dir>/.lock’) is acquired;
The file ‘/<cache_dir>/foo/1.0.0/af8d/.copying-a’ (or -b, -c etc) is created. This tells rez that this variant is being copied and cannot be used yet;
The file ‘/<cache_dir>/foo/1.0.0/af8d/a.json’ is created. Now another proc/thread can’t create the same local variant;
The file lock is released;
The variant payload is copied to ‘/<cache_dir>/foo/1.0.0/af8d/a’;
The ‘.copying-a’ file is removed.

Note that the variant will not be cached in the following circumstances, unless force is True:

The variant is not cachable as determined by Variant.is_cachable;
The variant is from a local package, and ‘config.package_cache_local’ is False;
The variant is stored on the same disk device as this cache, and config.package_cache_same_device’ is False.

Parameters:

variant (Variant) – The variant to copy into this cache
force (bool) – Copy the variant regardless. Use at your own risk (there is no guarantee the resulting variant payload will be functional).

Returns:

2-tuple: - str: Path to cached payload - int: One of VARIANT_FOUND, VARIANT_CREATED, VARIANT_COPYING, VARIANT_COPY_STALLED

Return type:

tuple

remove_variant(variant)#

Remove a variant from the cache.

Since this removes the associated cached variant payload, there is no guarantee that this will not break packages currently in use by a context.

Note that this does not actually free up associated disk space - you must call clean() to do that.

Returns:: One of: - VARIANT_REMOVED - VARIANT_NOT_FOUND - VARIANT_COPYING
Return type:: int

add_variants_async(variants)#

Update the package cache by adding some or all of the given variants.

This method is called when a context is created or sourced. Variants are then added to the cache in a separate process.

get_variants()#

Get variants and their current statuses from the cache.

Returns:

List of 3-tuple:

Variant: The cached variant
str: Local cache path for variant, if determined (’’ otherwise)
int: Status. One of: - VARIANT_FOUND - VARIANT_COPYING - VARIANT_COPY_STALLED - VARIANT_PENDING

Return type:

tuple

run_daemon()#

Run as daemon and copy pending variants.

Called via rez-pkg-cache –daemon.

clean(time_limit=None)#

Delete unused package cache files.

This should be run periodically via ‘rez-pkg-cache –clean’.

This removes:

Variants that have not been used in more than ‘config.package_cache_max_variant_days’ days;
Variants that have stalled;
Variants that are already pending deletion (remove_variant() was used).

Parameters:: time_limit (float) – Perform cleaning operations only up until this limit, resulting in a possibly incomplete cleanup. This is used to keep the cache size down without having to periodically run ‘rez-pkg-cache –clean’.