Currently, as users of alchemiscale generate results for their AlchemicalNetworks, results grow monotonically in the object store without any real bound. Although it is possible for an operator to manually delete result objects from the object store to mitigate this, this leaves behind stale references to results in the state store that can confuse users that try to access them.
We should create a new RetainerService that enforces a configured data retention policy. An MVP for this service could at the very least enforce a strict time-based policy, where ProtocolDAGResults and their ancillary artifacts (permanent files, logs, STDERR and STDOUT streams) older than the configured time window are deleted from the object store, and their corresponding ProtocolDAGResultRefs in the state store are updated to reflect this deletion.
A more sophisticated version of the above would be time-based deletion of ProtocolDAGResult data based on the most recent time any of their corresponding ProtocolDAGResultRefs were last accessed. This would keep results that are still being accessed consistently enough around, while removing results that are no longer relevant to users.
Currently, as users of
alchemiscalegenerate results for theirAlchemicalNetworks, results grow monotonically in the object store without any real bound. Although it is possible for an operator to manually delete result objects from the object store to mitigate this, this leaves behind stale references to results in the state store that can confuse users that try to access them.We should create a new
RetainerServicethat enforces a configured data retention policy. An MVP for this service could at the very least enforce a strict time-based policy, whereProtocolDAGResults and their ancillary artifacts (permanentfiles, logs,STDERRandSTDOUTstreams) older than the configured time window are deleted from the object store, and their correspondingProtocolDAGResultRefs in the state store are updated to reflect this deletion.A more sophisticated version of the above would be time-based deletion of
ProtocolDAGResultdata based on the most recent time any of their correspondingProtocolDAGResultRefs were last accessed. This would keep results that are still being accessed consistently enough around, while removing results that are no longer relevant to users.