[Feature]: Automate execution of complete run test

[chengzhuzhang]

How will this affect the next version number?

New feature (increment MINOR version)

Is your feature request related to a problem?

Similar to E3SM-Project/e3sm_diags#1084, for zppy we can fully automate zppy's testing workflow:

• Setup an automated job (e.g., cron) that runs the agent skill and flags unexpected diffs based on severity (the later was taken care by Rank image check failures by severity #865)
• We can even have the agent post on GitHub discussions for testing results automatically, with links to the HTML diffs page and a summary report.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[forsyth2]

for zppy we can fully automate zppy's testing workflow

#774 did the vast majority of this. Let's look at the automated test docs to see what remains to be done:

• Step 1: Determine what the current expected results are -- the script could be updated to "note the date the expected results were last updated."
• Step 2: Review changes since expected results were updated -- this might be a little trickier to automate, but could be done. We'd need to record merge commits made in each repo since a certain date.
• The automated test script -- the script itself has some setup, which we'd need to have set to some standard values for automated cron job testing. Tony had some notes on setting up cron jobs here.
• Step 8: the image checker test -- this is currently only manual because it requires launching on a compute node (for multi-processing), which was easy enough to do by hand without setting up a batch job. We could update the script to launch that automatically.
• "In your Markdown report:" -- We could have the script write out the whole markdown report to paste to the weekly testing log and maybe even post it there automatically.

The above can be condensed into 3 tasks:

1. Have the script write the Markdown report so I don't write it out myself.
2. Have the script auto-launch the image checker test.
3. Have the script auto-run every week with no human input.

There are also a few new and upcoming features that are helpful:

1. Allow multiple image checker runs on the same zppy output -- recently merged. This makes it easier to test changes/new features of the image checker test without re-running zppy itself.
2. Partial updates of expected results. I've already been using commits from this branch to do partial updates, but I think before merging we wanted to add some sort of tracking of environments used.
3. Rank image check failures by severity, as noted above, which Xylar has implemented. I'm in the process of testing it.

There's also a feature we decided against:

1. Testing with frozen envs as described in testing strategy discussion, Integration testing strategy for `pcmdi_diags` #848 and Test with frozen envs #849

Finally, there are simply more test cfgs and expected outputs we need to add:

1. Legacy 3.2.0 cfgs & EAMxx-included simulation

We can even have the agent post on GitHub discussions for testing results automatically, with links to the HTML diffs page and a summary report.

In general, I don't think we should be using AI agents if they're not actually necessary. In this case, essentially everything discussed could be computed deterministically by a script -- even posting on GitHub, which I'm sure has some sort of API call available.

AI agents would be more useful for tasks that simulate human thought, say writing a narrative summary of the key points of the diffs. I think the better use case of AI here would be to have it improve the deterministic script we already have.

I have a similar sentiment regarding the datathon project to auto-build zppy cfgs -- use AI to write a deterministic script and then use AI for some sort of qualitative/narrative overlay. That is -- we use deterministic, rule-based scripts until there simply aren't enough rules to follow, in which case AI agentic reasoning becomes more useful.

[tomvothecoder]

In general, I don't think we should be using AI agents if they're not actually necessary. In this case, essentially everything discussed could be computed deterministically by a script -- even posting on GitHub, which I'm sure has some sort of API call available.

AI agents would be more useful for tasks that simulate human thought, say writing a narrative summary of the key points of the diffs. I think the better use case of AI here would be to have it improve the deterministic script we already have.

Good point Ryan. Thanks for the feedback. I opened the initial issue to start the automation discussion in search of possible solutions.
I think it is important to automate the testing frameworks of both zppy and e3sm_diags as much as possible to alleviate time-consuming setup and run work. We'd only need to focus on debugging if issues appear.

I agree, we can setup most of the automation to be deterministic, including a cron job, running tests and comparison, and possibly an API call to GitHub with credentials to post summary reports on GitHub discussions to both notify and inform us.

[forsyth2]

#871 implements many of the points above. I will try it out next week, hopefully Monday, as part of the weekly testing.