Changing the default store

[utf]

Currently jobflow uses a memory store as the default store. This means that any outputs from workflows are lost at the end of the script or when a notebook is restarted.

What people think about changing the default store to be a MontyStore written to the ~/.jobflow folder? This is closer to what covalent does. It would mean that creating a basic atomate tutorial would also be very simple.

[JaGeo]

I think it is a good idea.

[davidwaroquiers]

I also think it would be beneficial. Now I would maybe still have a warning, e.g. when it gets to more than X jobs or flows, to tell users that they are maybe using something not tailored to high-throughput production.
Also pinging @gpetretto because there were some issues in MontyStore that prevent its usage for jobflow remote (not for jobflow from what I remember).

[gpetretto]

In my case, the issue with these file based mongoDB (MontyDB, mongita, tinymongo) is that they typically do not support find_one_and_update and plus other features like aggregations. In general this should not be an issue in this case, since the JobStore is used as a proper maggma Store. The problem is that in jobflow-remote I need to directly use the MongoDB methods in order to guarantee the state consistency and avoid complicated actions passing thorugh the Store.

On the other hand, if used with the fireworks manager there could be multiple fireworks trying to update the DB at the same time. I quickly checked but I could not find details about how MontyDB handles concurrency. In any case I expect that the MontyStore would need to be updated in order to lock the file before applying a change, otherwise I am afraid there will be inconsistencies.

Having looked into file and DB locking I should also add that, since several cluster use NFS, handling concurrency may be tricky. SQLite DBs may be corrupted for concurrent access on an NFS. I used flufl.lock in jobflow-remote for the lock of one file. It is defined as "NFS-safe", but I made a test trying to push it with high number of very fast access on a cluster where the FS is not very fast and it sometimes failed. I was fine with that, since I think it was an unlikley situation for my use case, but still to be taken into account as a source of potential issue when using file based DBs.

[JaGeo]

I think this issue might also be relevant here: materialsproject/maggma#832

[gpetretto]

Also the discussion about the MemoryStore may be useful: materialsproject/maggma#830. And the associated PR: materialsproject/maggma#846