clarify filtering documentation

[reidsunderland]

https://metpx.github.io/sarracenia/Reference/sr3_options.7.html#:~:text=In%20the%20above%20snippet,higher%20in%20the%20file..

After upgrading to 3.03, which includes a fix where Sundew extensions are now included in the URL when filtering, we came across a few configs with accept statements that no longer matched the URLs with Sundew extensions because the accepts ended in $.

The documentation also has a FIXME note:

The case we ran into was:

accept .*.csv$

which worked on 3.02 when URLs were http://something.com/2026.csv but not once the sundew extension was added, e.g. http://something.com/2026.csv:EXTENSION:....

---

The purpose of this issue is to:

1. Document clearly that Sundew extensions are included when filtering.
2. Resolve the FIXME
3. Potentially improve/clean up all parts of the sr3 Options document that relate to accept, reject and filtering

We use regex match:

re.match(pattern, string, flags=0)
If zero or more characters at the beginning of string match the regular expression pattern, return a corresponding Match. Return None if the string does not match the pattern; note that this is different from a zero-length match.

• https://docs.python.org/3/library/re.html#re.match

This means that .* is usually required at the beginning of an accept/reject statement, but not at the end. If you want to match something specific at the end of the line, you need to terminate with $.

accept .*csv
# accepts:
# ...something.csv
# ...something.csv_even_stuff_at_the_end

accept .*csv.*
# effectively the same as above

accept .*csv$
# only accepts ...something.csv
# not something.csv.with_stuff_at_the_end

This might be obvious to regex experts, but it is worth having it clearly documented for anyone who is not an expert, and for the sake of documenting the expected behaviour.

[robjarawan]

I went through this against current development and both implementations. The main 3.02 to 3.03 point is right, but I think we should tighten the issue before someone picks it up.

The examples should use .*\.csv because the unescaped . matches any character. I’d also call this the string used for accept/reject filtering rather than saying the URL itself changes. The Python flow conditionally appends :<sundew_extension> to that matching string; it doesn’t modify the notification URL.

The re.match explanation also needs to be scoped to Python components. cpost and cpump use POSIX regexec, match a pathname or relPath, and don’t append the separate Sundew header. Their leading .* behaviour is different.

I’d make “first matching rule wins” explicit and bound the work to the English and French copies of sr3_options.7.rst and the command-line guide. That gives the contributor a clear finish line. I wouldn’t ask them to mass-edit existing anchored configs because some anchors are intentional and the 3.03 upgrade guide already tells operators to audit them.

The existing #1573 tests cover the runtime change, so verification can be those focused tests plus a clean documentation build.

[petersilva]

It should say somewhere in the description of the matching method that, when present, the content of the sundew_extension header in a message is appended to the upstream URL.

Grandpa's Story Time

Originally (aka in Sundew) the filenames had colons in them, which was a huge issue on windows, and just really ugly and unhelpful everywhere else. So we tried to get rid of them entirely. It turned out people were using them, so we brought them back, but in a message field, to keep them out of the filenames whenever possible.

We brought in sundew_extension as a crutch for migration, but there was never any intent for them to perdure (French word... English escapes atm.) forever. I wonder if someone can get a good estimate of how many of our products are actually posted with their sundew_extension? It isn't supposed to be present everywhere, so it should not be assumed to be the universal default. People need to know about it because the biggest deployment happens to be a case where it is very often present.
But it's not a feature, it's a crutch to deal more easily with legacy.

For the specific example, I think you need to match a colon separator between the main url and the sundew_extension. So, to get the right spirit:

     accept .*\.csv:.*

You would want the colon after the .csv in the pattern to have a better chance of not false matching something in the extension or the rest of the pathname ( .csvhithere.txt:1:2: ) would match if you just have a trailing .* )

fwiw, the issue of colons in file names dates back to the original reasons for Sarracenia, and the relevant issues and feature requests are on sourceforge, because we never managed to migrate them.

Some history:

https://sourceforge.net/p/metpx/feature-requests/42/ "Decolonization of Sarracenia"

https://sourceforge.net/p/metpx/bugs/54/ "Sundew incompatibilities"

[reidsunderland]

The de-colonization one did get migrated: #39

I think most of our reliance on the Sundew extension is from other systems at this point. DMS publishes extensions as headers in the messages, then we download with filename NONE so the extension gets added to the filename. Sometimes they are required to differentiate between specific versions of products. E.g.:

...:DMS:CMC:WXAPP-AQHI_FCST:JSON:20260828204647
...:DMS:CMC:WXAPP-AQHI_FCST_NOTIFICATION_3_0:JSON:20260828204657

And a lot of the data received from CMOI includes an extension in the filename at the source. It also usually indicates the file type, e.g.:

MAPS_A0709C_CMC709_GLB_STD_A_250HPA_12Z:OCXCARTE:CMC:CHART:GIF size: 184.6KiB (189064)
MAPS_A0709C_CMC709_GLB_STD_A_250HPA_12Z:OCXCARTE:CMC:CHART:IMV6 size: 558.8KiB (572161)
MAPS_A0709C_CMC709_GLB_STD_A_250HPA_12Z:OCXCARTE:CMC:CHART:PNG size: 223.0KiB (228387)

We never add extensions when setting up new feeds, and we are also making an effort to avoid using them in filtering when possible. We are getting closer to retiring Sundew, so I think they will probably slowly go away over time, but they are still needed and useful in the short term at least. Having the clear documentation will be helpful.

It's probably worth noting in the documentation that we recommend writing accept/reject statements without using sundew extensions whenever possible.

[petersilva]

Would it be helpful to make appending sundew_extension to the match subject to a setting?

accept_pattern_includes_sundew_extension True|False

so it can be switched off, without having to lose the header. Might make it easier to omit from accept patterns.

[robjarawan]

I think this could help, but I’d keep it separate from #1769 since that PR is documenting the current behaviour.

Basically the setting would only control whether sundew_extension gets appended to the temporary accept/reject matching string. It wouldn’t remove the header or affect things like filename NONE.

I’d default it to enabled so existing filters keep working, and use a name that covers both accept and reject, something like filterIncludeSundewExtension. This would only apply to the Python flows since cpost and cpump don’t append the separate header.

I think it makes sense as a separate feature issue with tests for both settings, then we can update the documentation once the behaviour exists.

[reidsunderland]

I feel the opposite lol

I think it would add unnecessary complexity to have an extra option for it. Sort of related to Rob's note, it would also add confusion to how the filename option would work (does turning it on imply filename WHATFN all the time? or does filename NONE stop working when it's turned off?).

Most of the time when a Sundew extension is added, it's already in the filename (rather than a separate header), so it's "too late" to not use it for filtering.

If it were to be added, my opinion is to have it disabled by default (i.e. do not add the sundew extension when filtering), because that better reflects the eventual goal of Sundew extensions not existing. Ideally that option and related code would eventually be removed someday, so the default (off) should reflect that.

[petersilva]

I had forgotten about filename, of course that takes care of it.

[reidsunderland]

From #1769: there's a lot of duplication between the sr3 options and Command Line Guide documents.

There is a section in sr3 options that is trying to explain the ordering of options, but following the changes in #1769 will go into too much detail about accept/reject/filtering. The sequence #1 and sequence #2 example is also duplicated at both the top of the sr3 options document and in the section describing accept, reject and acceptUnmatched. There is more filtering info in the Command Line Guide, which is a huge and overwhelming document on its own.

I think it might be helpful to have a document dedicated to how filtering works.

In sr3 Options, we would keep:

• an explanation of ordering in the config file
  • this should include basic info about accept/reject, directory, filename, mirror, strip (i.e. everything that becomes a mask)
  • we should also note the newer ordering for subscriptions (expire, prefetch, broker, queueName, etc. has to go before subtopic)
• the section about accept/reject/acceptUnmatched, but simplify it and link to the filtering document for details about how it works, C vs. Python, etc.

In the Command Line Guide, we could also link to the Filtering document.

[petersilva]

The sr3_options file is a reference man page supposed to explain the file format. If you look at the earliest sections of the man page, it explains the ordering brieflly and includes a rudimentary discussion of accept/reject.

https://metpx.github.io/sarracenia/Reference/sr3_options.7.html

The purpose of a reference man page is to explain the details. So the overview at the top is sort of general, and serves as an introduction to the details that follow. When it comes to a section about accept and reject directives, it the convention around man pages is that it should be exhaustive. It should explain every possibility and nuance. That's what man pages are for. Reference man pages are not the place for simplified explanations, it's supposed to be like a dictionary... The simplified/tutotial/generalized stuff is what would go elsewhere.

[reidsunderland]

I'm okay with having all the details in sr3 Options. The CLI guide can link to that instead