trajectory formats for reading/writing

[franzlang]

Currently, MDMC has its own Trajectory object that gets created from the MD simulation. While there are some sparsely documented functions to pickle such an object (see https://github.com/MDMCproject/MDMCv0.2_pilot/blob/master/tests/test_data/MDMC_objects/README.md) and one example trajectory saved for testing (https://github.com/MDMCproject/MDMCv0.2_pilot/blob/master/tests/test_data/MDMC_objects/trajectory.zip), this is not in any common format and neither is it obvious how to re-import/load the pickled trajectory.

In the long run, we need to offer the possibility of saving and loading trajectories within MDMC. For this we need to decide on what formats we need to be able to read from and write to. The reasons why we would want to be able to read/write trajectories comprise:

• being able to save progress on an existing MDMC study and potentially continue it at a later stage
• testing MDMC against output from other software such as NMolDyn
• enable users to take the MDMC output and do run post-processing on it that we have not implemented yet (using e.g. MDAnse or NMolDyn)

We have been told that most MD engines have their own, unique formats that they use (some binary, some ASCII). In addition, the ASE Python module (already in use for reading from cif files and visualisation) can apparently read trajectories from a fair number of the popular MD engines and has its own format that it can save to.

Separately, it has also been suggested that MDAnalysis might be suitable for the task (#264 (comment)).

[alexhroom]

We mentioned pickling in the meeting where we discussed this - ASE seems to have moved away from it for security reasons, so it may be wise to rule that out as an option.

[iermi]

There is a nice page which contains all trajectory file formats. Here is the link to it: https://docs.mdanalysis.org/2.0.0-dev0/documentation_pages/coordinates/init.html
In general, any trajectory can be converted to pdb-format, but for a simulation with a length of several nanoseconds (if we have output often) such a format can cost a lot of memory.

[franzlang]

MDAnalysis has a webinar on the 12th October: https://bioexcel.eu/webinar-mdanalysis-interoperable-analysis-of-biomolecular-simulations-in-python-2021-10-12/
for those interested in attending

[iermi]

Another relevant page is the one about the xdr-file. https://docs.rs/crate/xdrfile/0.2.0
xtc-format is used also in lammps, but one has to do a certain compilation for that.

[franzlang]

the mdanalysis modules look very promising for reading and writing a number of trajectory file formats. while ASE also seems to support the reading of a fair few popular MD code's trajectory formats it is less explicit about the exact supported formats: https://wiki.fysik.dtu.dk/ase/ase/io/io.html

[thomasholmrod]

check this out: https://mdtraj.org/1.9.4/index.html - can also read hdf5!

[franzlang]

our DL_POLY colleagues also suggested to have a look at H5MD

[jfkcooper]

Some thoughts on this:

Trajectories are created by the MD engine (currently LAMMPS and DL_Poly). These can be interrogated by MDMC. The MD simulations can take an extremely long time so need to have some intermediate savepoints which can be resumed from. There is a choice of “who” makes these save points (i.e. MDMC or the MD engine), with pros and cons to each choice.

After discussion the current thought is that it is best to make the engines do their own saveing/resuming, since this is already built in. The drawback is of course that the engines do not all output to the same file format, and while it is not expected to switch engine mid-way through a run, it would have made some things easier.

Letting the engines do the saving also bypasses potential memory leaks which could occur doing it ourselves.
The outputs should all be some form of binary, since they are going to get large, but most engines know this and support binary outputs.
Currently LAMMPS has h5md supported https://docs.lammps.org/dump_h5md.html if LAMMPS is built with the H5MD package, but have restart files https://docs.lammps.org/write_restart.html built in and as the expected file for restarting a run.

DL_Poly writes to a HISTORY file and can again restart, with a little change to the CONTROL file https://www.ccp5.ac.uk/sites/www.ccp5.ac.uk/files/DL_POLY/data/training/MANUAL_USR.pdf section 8.2.4 (p137). This HISTORY file should probably be written in netCDF rather than ascii, but this also needs setting up in the CONTROL file.

[jfkcooper]

A further thought to this is the fact that there are multiple reasons for storing the trajectories, (as mentioned above), and resuming is only one of these. The post-run analysis may indeed warrant its own save file, external from the engine which created it. Where a format such as H5MD once again seems a good option. This probable deserves some discussion, but may split this task into two, one dealing with the intermediate files allowing trajectories to be resumed, and the other dealing with a final state file for analysis.

[alinelena]

netcdf in dlpoly was removed due to bit rotting. the plan is to move to hdf5 using h5md specification.

[jfkcooper]

@alinelena Am I vaguely correct about the two file types for DL_Poly (for intermediate saving with an aim to resume, and for final output) or would the h5md support both?

[alinelena]

yes there are two files for resume. 1 is containing positions, forces and velocities. this can be safely put in hdf5 format. the other REVIVE, used to be called in the past, now you can use whatever name you want, contains stats about various observables. This is a little bit more tricky but not impossible