fileOp causing problems in polls, watches

[reidsunderland]

Watches and polls post fileOp messages by default (e.g. renames, mkdirs, links, delete), but this is usually unexpected and surprising, especially when downloading to a cluster.

I think we need a convenient, built in way to suppress fileOp messages and publish normal "file download" messages instead.

For example, a watch. A user uploads files named SOMETHING.TMP then renames to SOMETHING. The watch publishes two messages:

1. a normal download mesages for SOMETHING.TMP
2. a rename event to rename SOMETHING.TMP to SOMETHING

This is confusing outside of mirroring. The SOMETHING.TMP message fails to download (because SOMETHING.TMP is renamed and gone by the time the sarra tries to download it). And the rename message causes an error, because SOMETHING.TMP didn't get downloaded (even if it did, there's a good chance it would have been downloaded to another node). The rename message should fall through to download, so the file isn't lost, but the errors cause a lot of confusion.

Similar with links, when we are polling or posting files, usually we just want to receive a copy of the file. We don't want a link on our cluster, we want the actual file.

Posting directories (mkdir events) are mostly harmless, but they cause errors when v2 subscribers/sarras try to download them.

Changing the fileEvents option doesn't solve all of these cases. We usually use fileEvents create,modify because we need to detect creations and modifications (existing file gets overwritten or updated), but this causes renames to be posted.

[reidsunderland]

We are currently using this plugin as a workaround:

import logging
from sarracenia.flowcb import FlowCB
from sarracenia.flowcb.log import Log

logger = logging.getLogger(__name__)

class Delete_fileop(FlowCB):

    """
    In certain cases, fileOps cause undesirable behaviour.

    For example, if you post a symlink, sr3 will post a symlink event. If the target of the symlink does not already
    exist on the destination, you'll be left with a broken symlink at the destination. This is what we want in the
    HPC mirroring use case, where the destination should mirror the source as closely as possible.

    Other options exist to change this behaviour, such as ``follow_symlinks``, which would post 1 symlink message and
    1 file message. This doesn't always work because the subscriber might:
      a) put the file in a different directory, based on the directory option(s) in the subscribing config,
         so the symlink message would still cause a broken symlink to be created
      b) be running on a cluster, where the link and file download messages may be processed by different nodes

    Sometimes we just want to download a file. We don't care that it's a symlink at the source.

    This plugin should work in either the posting config or the subscribing config.
    """

    def __init__(self, options, logger=logger):
        super().__init__(options,logger)
        self.flowcb_log = Log(options)

    def after_accept(self, worklist):
        for msg in worklist.incoming:
            changed = False
            before = self.flowcb_log._messageAcceptStr(msg)

            if 'fileOp' in msg:
                fileOp = msg.pop('fileOp', None)
                logger.debug(f"removed fileOp {fileOp}")
                changed = True
            if 'post_fileOp' in msg:
                fileOp = msg.pop('post_fileOp', None)
                logger.debug(f"removed post_fileOp {fileOp}")
                changed = True
            if '_deleteOnPost' in msg:
                msg['_deleteOnPost'].discard('fileOp')
                msg['_deleteOnPost'].discard('post_fileOp')

            if changed:
                after = self.flowcb_log._messageAcceptStr(msg)
                logger.info(f"changed ({before}) to ({after})")

One possible solution is adding this plugin to the sr3 repo. It might be useful to add an option that lets a user define which fileOps get deleted (e.g. renames and links, but allow mkdir and delete)

[petersilva]

I would think that filaAgeMin helps. the .tmp thing should mean files don´t get very old without being written to... so something like 5 seconds, means the .tmp files never get posted. If it posts a rename fileOp, and the source file was never downloaded, normal download logic should just fall through to a download.

[petersilva]

also reject *.tmp

[petersilva]

I mean, I guess this is obvious, but just in case: The assumption here is that the thing doing the writing is NOT sarracenia. IF it is Sarracenia, then best practice would be to not write a .tmp file (tmpfile None), not use a poll, have the sender post the delivery notice from the process that wrote the file after delivery to a consuming exchange, completely avoiding any polling, and/or tmp file write & rename delay.

[reidsunderland]

There was a lot wrong with the watch config that made me create this issue, so it was hard to know exactly why we started having problems with it. It was migrated from a poll and the old sleep setting was left in there (sleep 90) which may have also contributed to the problems.

It does already have fileAgeMin 30s, and it had reject .*\.TMP$, which stopped working when the Sundew extension was added. After changing the reject, the create events for the .TMP files were being rejected correctly.

At that point the messages being posted were only renames:

sarracenia.flowcb.log after_post posted to amqp://broker,exchange,v02.post a rename /apps/users/.../FILE.grb.bz2.TMP with baseUrl: ftp://user@server relPath: FILE.grb.bz2

And the sarra did fall through to download:

2026-08-30 22:06:55,530 [INFO] sarracenia.flowcb.log after_accept accepted: (lag: 360.25 )  exchange: xs_EXCHANGE subtopic:  a rename /apps/users/username/FILE.grb.bz2.TMP with baseUrl: ftp://user@server relPath: FILE.grb.bz2 sundew_extension: pull-depot-SUBDIR:FDSA:SAT:3:BIN id: IKFEKBg size: 7.6MiB (7981302)
2026-08-30 22:06:56,725 [INFO] sarracenia.flow renameOneItem old file /apps/users/username/FILE.grb.bz2.TMP not found, can't rename to /apps/sarra/public_data/20260830/SOURCENAME/SUBDIR/22/FILE.grb.bz2
2026-08-30 22:07:03,574 [INFO] sarracenia.flowcb.log after_work renamed ok: /apps/sarra/public_data/20260830/SOURCENAME/SUBDIR/22/FILE.grb.bz2 size: 7.6MiB (7981302) rate: 3.1MiB/s (3294639.40 B/s)
2026-08-30 22:07:03,579 [ERROR] sarracenia.flowcb.v2wrapper sumstrFromMessage unknown fileOp: {'rename': '/apps/users/username/FILE.grb.bz2.TMP'}
2026-08-30 22:07:03,584 [INFO] sarracenia.flowcb.log after_post posted to amqp://feeder@localhost/,xpublic,v02.post.20260830.SOURCENAME.SUBDIR.22 a rename /apps/users/username/FILE.grb.bz2.TMP with baseUrl: http://server.ca/ relPath: 20260830/SOURCENAME/SUBDIR/22/FILE.grb.bz2 size: 7981302 id: IKFEKBg

But I still think it's confusing to be getting rename messages when the goal is not really to rename a file but download it, and the sarra re-publishes the message as a rename to downstream consumers. And the after_work log also says "renamed ok", when the file was really downloaded, not renamed.

There's also that v2wrapper error in the logs, which seems like it should not be happening based on the code:

sarracenia/sarracenia/flowcb/v2wrapper.py

Lines 61 to 83 in 446a826

	if 'fileOp' in msg:
	if 'rename' in msg['fileOp']:
	msg['oldname'] = msg['fileOp']['rename']
	if 'link' in msg['fileOp']:
	hash = sha512()
	hash.update( bytes( msg['fileOp']['link'], encoding='utf-8' ) )
	sumstr = f'L,{hash.hexdigest()}'
	elif 'remove' in msg['fileOp']:
	hash = sha512()
	hash.update(bytes(os.path.basename(msg['relPath']), encoding='utf-8'))
	sumstr = f'R,{hash.hexdigest()}'
	elif 'directory' in msg['fileOp']:
	hash = sha512()
	hash.update(bytes(os.path.basename(msg['relPath']), encoding='utf-8'))
	if 'remove' in msg['fileOp']:
	sumstr = f'r,{hash.hexdigest()}'
	else:
	sumstr = f'm,{hash.hexdigest()}'
	else:
	logger.error(f"unknown fileOp: {msg['fileOp']}" )
	return sumstr

That could be a bug, I'll look into that more to try to understand why it's happening.

[reidsunderland]

The misleading unknown fileOp error message for renames was an easy fix, #1771