Centralize `.tigerflow` at home directory

[yoonspark]

Currently, .tigerflow is created within each pipeline output folder, resulting in some issues:

• With these internal metadata folders scattered across random folders, it becomes difficult for users to trace and manage many pipelines in a systematic way. Currently, they need to remember every pipeline's output folder.
• It is easy for users to poke around and accidentally meddle with internal data in .tigerflow (especially as they need to go inside the folder to check log and error files), which may affect the pipeline's tracking mechanism.

We may remedy these limitations by organizing all pipelines' internal data into a single .tigerflow at home directory (i.e., ~/). That is, each "unique" pipeline gets its own subfolder under this centralized .tigerflow, where its internal data (e.g., intermediate files) are placed and managed.

With this separation of pipeline metadata from output folder, we may redefine the meaning of a "unique" pipeline that can be resumed: we can now base a pipeline's identity on its config file (YAML can be converted to a canonical representation with essential information only; and then be hashed) rather that its output folder. This opens up interesting new possibilities:

• The same pipeline (i.e., a pipeline with identical config) can be resumed on any pair of input and output folders.
• Multiple instances of the same pipeline (i.e., a pipeline with identical config) can be run simultaneously.
• Different pipelines (i.e., pipelines with different configs) can target the same output folder.

These become possible because all pipeline runs with the same config would share the same internal data, regardless of input and output folder of each run. (Note: We should think through race conditions among simultaneous runs of the same pipeline.)

Regarding log and error files, which users need to check for debugging, we can place their symlinks in the output folder. This way, we do not duplicate these files while enabling users to check them from within output folder they designated. This works because symlinks can be moved around and even deleted without affecting originals. Since the originals are all put in a fixed hidden location (i.e., ~/.tigerflow/) that users would not easily meddle with, broken symlinks are unlikely. The only potential issue is that contents in the original can be modified via its symlink, but it is a tolerable risk given users have little motivation to do so (and, if they did do, they should have a good reason, which should be reflected in the original). Hence, the output folder would be populated like so:

outputs/
└── <task>/
    ├── logs/      # Symlinks
    ├── errs/      # Symlinks
    ├── 1.txt      # Real copy
    └── ...

Interface

Concretely, running

tigerflow run [OPTIONS] CONFIG_FILE INPUT_DIR OUTPUT_DIR

would first check if the given config hashes to a value that already exists under the centralized .tigerflow. If not, it will create a new one, like so:

~/.tigerflow/
└── a3f5c8e2b17d4a906f1c7e5b9d2a8f34c6e1b7d9035a2f8c4e6d1a9b7f3c5e82/
    ├── .symlinks/        # Input file tracking
    ├── .finished/        # Completion markers
    └── <task>/           # Task outputs (intermedidate)

where a3f5c8e... is the deterministic hash value corresponding to the given pipeline config (more precisely, its canonical representation with non-essential information stripped out).

If the hash already exists, the command simply resumes the corresponding pipeline based on the existing internal data.

Users could also explicitly resume an existing pipeline with:

tigerflow resume [OPTIONS] PIPELINE_HASH INPUT_DIR OUTPUT_DIR

where PIPELINE_HASH can be obtained via:

# List all existing pipeline IDs (hashes)
tigerflow ls

# Print config corresponding to the pipeline ID (hash)
tigerflow preview PIPELINE_HASH

[yoonspark]

Colin shared valid feedback that the hashing-based design proposed above may result in incorrect pipeline resumption. For instance, researchers may want to run the same pipeline with different parameters (#114), in which case each experimental run should have its own progress tracking. The hash-based design cannot support this as it forces all these experimental runs to share the same progress tracking.

Based on this and other feedback, I propose a new design as follows (planned for team discussion with @cswaney and @Nina-mvH later).

Running

tigerflow run [OPTIONS] CONFIG_FILE INPUT_DIR OUTPUT_DIR

should create a fresh new pipeline subfolder under ~/.tigerflow/, even if repeated exactly same, like so:

~/.tigerflow/
└── f47ac10b-58cc-4372-a567-0e02b2c3d479/
    ├── run.pid           # PID of pipeline process (also serves as lock)
    ├── run.log           # stdout/stderr of pipeline process
    ├── .symlinks/        # Input file tracking
    ├── .finished/        # Completion markers
    └── <task>/           # Task outputs (intermedidate)

where f47ac10b... is the pipeline's unique ID (generated via UUID).

Note that we retain the existing locking mechanism (i.e., run.pid) so each pipeline can be accessed and executed by a single process only. This prevents the risk of racing conditions.

The pipeline can be resumed using its ID, like so:

tigerflow resume [OPTIONS] PIPELINE_ID

where runtime details (e.g., input/output dir, resources for each task) could be overwritten optionally. By default, the pipeline resumes with the same runtime details as its latest run.

Users can discover and manage existing pipelines using the following commands:

# List all existing pipeline IDs
tigerflow ls

# Print config corresponding to the pipeline ID
tigerflow preview PIPELINE_ID

[yoonspark]

Discussed the new design above with @cswaney and @Nina-mvH, who are on board with it overall. We will examine its details more critically in our next team meeting — to avoid costly surprises.

Some good suggestions were made during the meeting:

• We should assign ID to each "run" of the pipeline (e.g., the second resumed run of pipeline f47ac10b-58cc-4372-a567-0e02b2c3d479) so it can be identified with the specific config and other runtime details (e.g., input/output dir) if users want to inspect or reproduce previous runs. For uniqueness, each run ID can be generated by concatenating its PID with its start time (e.g., seconds elapsed since UNIX epoch).
• We should keep the record of all prior run's details including its specific config used and output/input dir targeted. They can be keyed by the run ID discussed above so we can enable crosswalk later.
• We should enable users to use a different location for storing the centralized .tigerflow. We can parametrize it with an environment variable that defaults to home (~).