Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs/design/git-annex-content.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@ dereferenced (dereferencing would give the object path, identical for all
files with the key). Results are then only shared by files at the same path,
which is needed whenever the result depends on the path, e.g. on the extension
(the key's extension may differ from the file's) or on neighboring files.
Pairing is not *sufficient* for the latter, though: it keeps results separate
per location, but does not make them change when a neighbor does (see
[Other files the function reads](#other-files-the-function-reads)).
The path is made absolute with `os.path.abspath`, which does not resolve
symlinks (neither the file's nor its parent directories'), so the same file
reached through different paths gets separate entries; that errs on the safe
Expand All @@ -95,6 +98,28 @@ With `pair_with_path=False`, the fingerprint is the key alone, and results are
shared by all files with the same content: twins in a dataset, and the same
file across clones. Only use it for results that depend on the content only.

## Other files the function reads

**Only the first argument is fingerprinted**, in `stat()` mode and with a
custom fingerprint alike. If the decorated function also reads other files,
e.g. the BIDS sidecar `sub-01_bold.json` of `sub-01_bold.nii.gz`, or metadata
inherited from parent directories, changes to those files do not change the
cache key, so the stale result is returned. This predates custom
fingerprints and is not specific to git-annex; `pair_with_path` does not
change it, as the pair `(path, key of the .nii.gz)` stays the same when only
the sidecar changes.

To have such changes noticed, either:

- fold fingerprints of the relevant files into a custom fingerprint, e.g.
`(annex_key_fingerprint(path), *(fingerprints of the sidecars))`, keeping in
mind that the callable runs on every call and must stay cheap, and that it
must change whenever the result may (no "modified just now" window applies);
or
- have the caller pass something that identifies the content of those files
(e.g. their keys or `stat()`s, or the parsed sidecar itself) as extra
arguments of the decorated function, which are part of the cache key.

## Dropped content

With `stat()`, a locked file whose content was dropped is a broken symlink: it
Expand Down
4 changes: 3 additions & 1 deletion src/fscacher/annex.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ def annex_key_fingerprint(path, *, pair_with_path=True):
(as given, not dereferenced) and the key, so results are only shared by
files at the same path, as needed when they depend on the path (e.g., on
the extension or on neighboring files). If false, results are shared by
all files with the same key, e.g., across clones of a dataset.
all files with the same key, e.g., across clones of a dataset. Either
way, only this file is fingerprinted: changes to neighboring files (e.g.,
sidecars) do not invalidate cached results.
"""
try:
path = os.fsdecode(os.fspath(path))
Expand Down
6 changes: 6 additions & 0 deletions src/fscacher/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,12 @@ def memoize_path(self, f=None, *, exclude_kwargs=None, custom_fingerprint=None):
`None` to fall back to ``stat()``. For a directory, it may return a
fingerprint of the whole tree, or `None` to fingerprint it by
``stat()``-ing each file as usual.

Only the first argument is fingerprinted, with or without
``custom_fingerprint``: changes to other files the function reads
(e.g., sidecar files next to it) do not invalidate cached results.
Pass something identifying their content as extra arguments, or fold
their fingerprints into a ``custom_fingerprint``, to have them noticed.
"""
if f is None:
return partial(
Expand Down
Loading