Revamp Run Scripts

[emerali]

• Add option to save generated configs for ML purposes (should just throw them out by default)
• Add "bin" option: kinda like "skip" but averages over K measurements to produce a single measurement
• SIGTERM handling
  • catch sigterm and checkpoint immediately
  • doesn't matter if we're in the middle of cluster construction, when we load the chain back up again we'll start with a full diagonal update anyway
  • could be bad if the sigterm happens in the middle of a cluster flip though...
    • maybe instead of "flipping" we "set" the legtype to a value that was sampled, that way we can just re-run the cluster_update! fn after loading up the checkpoint to make sure everything's consistent
    • wait but how do we know if it was interrupted during link list construction or the cluster update?
    • possible solution: keep track of which update step we're currently performing, if job is interrupted, we know which step to redo
• Checkpoint at regular time intervals (+ sigterm catching)
• Checkpoint file should save both the QMC state as well as the PRNG state
• Add function to restart a simulation from a checkpoint
• For (L)TFIM, allow reading interaction matrix from a file

[isaacdevlugt]

Been having problems with this lately. Specifically, the .jld2 file will save, but the program will get sigterm'd before the .csv is saved. For example, let's say batch=400 completed and we're on to the saving part of the run script. The batch=400 .jld2 file is saved, but then we get sigterm'd. When we go to continue the simulation from the batch=400 QMC state, the next batch is 401. However, in the .csv file, there is a "jump" between batch=399 and batch=401 (after batch 401 is completed).

Working on a bandaid-type fix.

[isaacdevlugt]

I think a quick fix is to change the following in the run scripts (in the continue_simulation function):

batches = sort(batches, by=x->parse(Int, x), rev=**false**)

With rev=false instead of rev=true, the code first tries older state files that haven't been deleted. So, if batch 400 was completed, and let's say the .csv file was NOT saved (sigterm'd), this also means that the state file associated to batch 399 was not deleted. Therefore, the simulation continues from batch 399.

There are better ways of ensuring that we don't "waste" batches. For example, we could have batches = sort(batches, by=x->parse(Int, x), rev=**true**) still, and then also read the .csv file to see the last batch it recorded. If it's one batch off, simply write the new batch of data into the .csv. However, the .csv files are typically quite large... I'm not sure how efficient it is to read through its entirety and gather the required information. It seems like a more simple fix is to just start from the previous batch (batch 399 in this example).

[emerali]

However, the .csv files are typically quite large... I'm not sure how efficient it is to read through its entirety and gather the required information.

it might be possible to avoid reading the entire file into memory and just count the number of lines instead

[isaacdevlugt]

it might be possible to avoid reading the entire file into memory and just count the number of lines instead

Tru