PACE sorted mdocs differ from SerialEM mdocs

[alisterburt]

Hey @eisfabian - thanks for all of your hard work here!

I've heard a couple of rumblings now that the 'sorted' mdoc files PACE writes out here

PACEtomo/PACEtomo.py

Lines 261 to 280 in 90d5ad2

	def writeMdoc(header, items, filename):
	content = ""
	for line in header:
	if line.startswith("[T"):
	content += os.linesep
	content += line + os.linesep
	content += os.linesep
	for i, item in enumerate(items):
	content += "[ZValue = " + str(i) + "]" + os.linesep
	item.pop("ZValue")
	item.pop("index")
	for key, attr in item.items():
	content += key + " = "
	for val in attr:
	content += str(val) + " "
	content += os.linesep
	content += os.linesep
	with open(filename, "w", newline="") as f:
	f.write(content)
	return

differ enough from SerialEM's files that Warp 2.0 fails to parse them correctly (e.g. warpem/warp#310)

I could write extra compatibility code but it might avoid other problems elsewhere to make the files here better match those produced by SerialEM, what do you think?

[eisfabian]

Hi Alister,

Thanks! I did have problems with AreTomo3 as well. But alignframes worked fine with the mdocs.
Do you know which part does not conform with the SerialEM mdoc files?

I know right now there are a few too many linebreaks in the header, but other than that I don't see what could cause the issues.

Thanks again and best wishes,
Fabian

[alisterburt]

@eisfabian thanks for the quick reply! I just compared two files, the differences I saw were

• trailing whitespace on each line within each mdoc section
• extra spacing in the DateTime entry i.e. SerialEM: DateTime = 23-Nov-2024 00:19:58 PACE: DateTime = 23-Nov-2024 00:19:58

The Warp datetime parsing is robust to the extra spacing in the timestamp and the trailing whitespace on that line so I'm still not sure why exactly your files break it 🤷🏼‍♂️

[eisfabian]

Thanks, Alister!

Could you maybe test it with this one?

def writeMdoc(header, items, filename):
    content = ""
    for line in header:
        if line.startswith("[T"):
            content += os.linesep
        content += line + os.linesep
    content += os.linesep
    for i, item in enumerate(items):
        content += "[ZValue = " + str(i) + "]" + os.linesep
        item.pop("ZValue")
        item.pop("index")
        for key, attr in item.items():
            content += key + " = " + (" ".join(attr) if key != "DateTime" else "  ".join(attr)) # Date and time is separated by double space in SerialEM
            content += os.linesep
        if i < len(items) - 1: # Don't add additional linebreak at end of file
            content += os.linesep
    with open(filename, "w", newline="") as f:
        f.write(content)
    return

I also removed an additional linebreak at the end of the file, but I can't imagine that this should make a difference.
If you find anything else, please let me know!

Best,
Fabian