From a3b609317e2c7524d86fdbc4c5cbd331981a5e5b Mon Sep 17 00:00:00 2001 From: Chris Blanton Date: Tue, 30 Jun 2026 17:04:45 -0400 Subject: [PATCH 1/5] Rewrite postprocess/analysis docs; fix Run FMS model placeholder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace scratchpad developer notes in postprocess.rst with complete prose and YAML examples for timeseries, climatologies, statics, surface masking, refineDiag, preAnalysis, and legacy cshell analysis - Remove reference to abandoned analysis-scripts repo in intro - Add preAnalysis scripts section (new, distinct from refineDiag) - Rename "Legacy refineDiag scripts" → "refineDiag scripts" - Rename "Analysis scripts" → "Legacy cshell analysis scripts" - Remove NEEDS UPDATING TODO from tools/pp.rst; add pointer to usage guide - Replace stale 2025 placeholder in Run FMS model with 2026.03 release note and link to legacy frerun documentation Co-Authored-By: Claude Sonnet 4.6 --- docs/tools/pp.rst | 4 +- docs/usage.rst | 6 +- docs/usage/postprocess.rst | 133 +++++++++++++++++++++++++++++++++---- 3 files changed, 125 insertions(+), 18 deletions(-) diff --git a/docs/tools/pp.rst b/docs/tools/pp.rst index 8190102b7..15654c053 100644 --- a/docs/tools/pp.rst +++ b/docs/tools/pp.rst @@ -1,6 +1,4 @@ -.. NEEDS UPDATING #TODO - - +For conceptual background and step-by-step workflow instructions, see the "Postprocess FMS history output" section in the Usage guide. ``all`` ------- diff --git a/docs/usage.rst b/docs/usage.rst index d6064b6aa..fb42788a8 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1,7 +1,7 @@ ===== Usage ===== -Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Please note that model running is not yet supported in FRE 2025; continue to use FRE Bronx frerun. +Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Model running is slated for the FRE 2026.03 release; continue to use FRE Bronx frerun in the meantime. YAML Framework ============== @@ -34,7 +34,9 @@ Build FMS model Run FMS model ============= -Check back in the latter half of 2025 or so. +Running experiments via fre-cli is slated for the FRE 2026.03 release. In the meantime, continue +to use the legacy FRE Bronx ``frerun`` tool; documentation is available at the +`FRE Bronx running an experiment `_ page. Postprocess FMS history output ============================== diff --git a/docs/usage/postprocess.rst b/docs/usage/postprocess.rst index 60d3cc651..f3657e73f 100644 --- a/docs/usage/postprocess.rst +++ b/docs/usage/postprocess.rst @@ -1,6 +1,6 @@ ``fre pp`` regrids FMS history files and generates timeseries, climatologies, and static postprocessed files, with instructions specified in YAML. -Bronx plug-in refineDiag and analysis scripts can also be used, and a reimagined analysis script ecosystem is being developed and is available now (for adventurous users). The new analysis script framework is independent of and compatible with FRE (https://github.com/NOAA-GFDL/analysis-scripts). The goal is to combine the ease-of-use of legacy FRE analysis scripts with the standardization of model output data catalogs and python virtual environments. +User plug-in scripts — refineDiag, preAnalysis, and legacy cshell analysis scripts — can be configured to run before and after postprocessing; see the dedicated sections below. In the future, output NetCDF files will be rewritten by CMOR by default, ready for publication to community archives (e.g. ESGF). Presently, standalone CMOR tooling is available as ``fre cmor``. @@ -165,25 +165,132 @@ Optional configuration (i.e. if xy-regridding is desired) Timeseries ---------- -Timeseries output is the most common type of postprocessed output. +Timeseries output is the most common type of postprocessed output. Each timeseries file contains +one variable from one postprocess component, spanning the requested date range. The temporal chunk +length — how many years of data to combine into one file — is configured globally in +``postprocess.settings.pp_chunks`` as an ISO8601 duration: + +.. code-block:: console + + postprocess: + settings: + pp_chunks: + - P5Y + +Timeseries files are organized under ``pp//ts///``. Climatologies ------------- -annual and monthly climatologies -less fine-grained than bronx -per-component switch coming -now it's one switch for entire pp +Climatologies are time-averaged files. All variables from a component appear in one file, with one +time-averaged level per file. Climatology output is configured per-component under a +``climatology:`` list, with ``frequency`` (``yr`` for annual, ``mon`` for monthly) and +``interval_years`` specifying the averaging period length: + +.. code-block:: console + + postprocess: + components: + - type: atmos_month + climatology: + - frequency: yr + interval_years: 2 + - frequency: mon + interval_years: 2 + +Annual climatologies produce one annually-averaged file per interval. Monthly climatologies produce +twelve monthly-mean files per interval. Statics ------- -underbaked, known deficiency -currently, takes statics from "source" history files - -Analysis scripts ----------------- +Static fields are time-invariant diagnostics such as grid geometry, land/sea masks, and +topography. They are extracted from the nominated source history files and placed in +``pp//ts/static/1yr/``. Surface masking for FMS pressure-level history ---------------------------------------------- +Pressure-level atmosphere history files contain data at all configured pressure levels, including +levels below the Earth's surface or above the model top. These out-of-range values are unphysical +and should be masked. Enable surface masking with the following switch: + +.. code-block:: console + + postprocess: + switches: + do_atmos_plevel_masking: True + +When enabled, atmosphere pressure-level variables are masked below the surface pressure and above +the model top. + +refineDiag scripts +------------------ +refineDiag scripts are user-provided cshell scripts that run on raw history files before +postprocessing and produce new "spoofed" history files. These new files are then postprocessed +alongside native model output, allowing users to define custom derived diagnostics as if they were +native model output. + +Configure refineDiag scripts under ``postprocess.refinediag``: + +.. code-block:: console + + directories: + refined_history_dir: /path/to/refined/history/output + + postprocess: + refinediag: + my_script_label: + script: /absolute/path/to/script.csh + inputs: + - atmos_month + outputs: + - atmos_month_refined + do_refinediag: true + +Output file names must differ from any existing history file names. Multiple refineDiag scripts +can be defined, each under its own label. + +preAnalysis scripts +------------------- +preAnalysis scripts are user-provided cshell scripts that run on raw history files before +postprocessing, similar to refineDiag scripts, but do not produce new history files. They are +suited for user-specific tasks such as populating custom databases or producing output outside the +FRE managed directory structure. + +Configure preAnalysis scripts under ``postprocess.preanalysis``: + +.. code-block:: console + + postprocess: + preanalysis: + my_preanalysis_label: + script: /absolute/path/to/script.csh + inputs: + - atmos_month + do_preanalysis: true + +Legacy cshell analysis scripts +------------------------------- +Legacy cshell analysis scripts are user-provided scripts that run after postprocessing using +postprocessed output to produce figures or other user-specified output. Unlike refineDiag and +preAnalysis scripts, analysis scripts have no downstream dependencies in the FRE postprocessing +workflow. + +Analysis scripts are configured under the top-level ``analysis:`` key, separate from +``postprocess:``: + +.. code-block:: console + + directories: + analysis_dir: /path/to/analysis/scripts + + analysis: + my_analysis_label: + legacy: + script: /path/to/analysis_script.csh + required: + data_frequency: mon + date_range: ['1979', '2020'] + +Each analysis script is associated with one data frequency. For scripts that span multiple +components, associate the script with the component that has the longest run interval. -Legacy refineDiag scripts -------------------------- +A new Python-native analysis framework is being developed for the FRE 2026.02 release. From a191511bfd78d0960612f458d4e5a1c695569585 Mon Sep 17 00:00:00 2001 From: Chris Blanton Date: Tue, 30 Jun 2026 18:46:24 -0400 Subject: [PATCH 2/5] Restructure docs to mirror FRE Bronx site layout Reorder nav (FRE tools before Usage), fold container into Setup, rename sections to match old structure (Yaml files, Build a model, Running an experiment, Postprocessing, User plugins), add About/Schema/ Variables/Directories subsections to Yaml files, expand postprocess with online/offline PP, sub-year, pressure levels, gap-filling, and validation sections, extract user plugin docs to user_plugins.rst. Co-Authored-By: Claude Sonnet 4.6 --- docs/container.rst | 48 +-------- docs/index.rst | 3 +- docs/setup.rst | 5 + docs/tools.rst | 6 +- docs/usage.rst | 88 ++++++++++++--- docs/usage/postprocess.rst | 209 ++++++++++++++++++++++-------------- docs/usage/user_plugins.rst | 77 +++++++++++++ 7 files changed, 294 insertions(+), 142 deletions(-) create mode 100644 docs/usage/user_plugins.rst diff --git a/docs/container.rst b/docs/container.rst index aae515545..e3e0446c2 100644 --- a/docs/container.rst +++ b/docs/container.rst @@ -1,51 +1,9 @@ Post-Processing Container ========================= -Previously, many GFDL workflows and configurations have only been accessible on gitlab. This is disadvantageous for outside collaboration, flexibility, community development. While the FRE workflow can now be conda installed, another deployment method of containerization has been developed. Containerzation of the FRE workflow at GFDL bolsters portability while also simplifying the environment set-up for the user. With the environment set-up done through the container build and runscript, this post-processing container work allows for more effective sharing of the workflow. - -BUILDING --------- - -In order to build the container, the user needs to have podman access on gaea. If needed, put in a helpdesk ticket. - -Files used to build container: - - Dockerfile-ppp - - cylc-flow-tools environment yaml - - runscript.sh - -The container will house the fre-cli tools and subtools, and any necessary packages needed for those tools. - -Using podman and apptainer to build, follow these steps: - -1. Clone the HPC-ME repository - -.. code-block:: console - - git clone git@gitlab.gfdl.noaa.gov:fre/HPC-ME.git - -2. Navigate into the ppp folder - -.. code-block:: console - - cd HPC-ME/ppp - -3. Build a container image - -.. code-block:: console - - podman build -f Dockerfile-ppp -t 2025 - -4. Save the image to a local tar file (It is recommended to name the container after the post-processing experiment name) - -.. code-block:: console - - podman save -o [name of container].tar localhost/2025 - -5. Create the singularity image file (sif) from the tar file - -.. code-block:: console - - apptainer build --disable-cache [name of container].sif docker-archive://[name of container].tar +The FRE post-processing container packages fre-cli and its dependencies into a portable +Apptainer/Singularity image, enabling postprocessing on systems where a full conda install is +not practical. The container is distributed as a pre-built ``.sif`` file. SETUP ----- diff --git a/docs/index.rst b/docs/index.rst index be868d80b..4d9259418 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,9 +16,8 @@ Welcome to ``fre-cli``'s documentation! what-is-fre setup - usage tools - container + usage api for-developers glossary diff --git a/docs/setup.rst b/docs/setup.rst index 55746b776..1556c9c28 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -45,3 +45,8 @@ Assuming one has ``conda`` in their path, then do the following:: # optional: load fre-nctools into your PATH to gain access to regridding and certain time-averaging routines # module load fre-nctools + +Using the container +=================== + +.. include:: container.rst diff --git a/docs/tools.rst b/docs/tools.rst index 655028ee8..c39e27265 100644 --- a/docs/tools.rst +++ b/docs/tools.rst @@ -1,8 +1,8 @@ .. last updated May 2025 -===== -Tools -===== +========= +FRE tools +========= ``fre`` ======= diff --git a/docs/usage.rst b/docs/usage.rst index fb42788a8..b0b757fcc 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1,11 +1,73 @@ ===== Usage ===== -Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Model running is slated for the FRE 2026.03 release; continue to use FRE Bronx frerun in the meantime. +Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model and ``fre pp`` +postprocesses the history output and runs diagnostic analysis scripts. Model running is slated for +the FRE 2026.03 release; continue to use FRE Bronx frerun in the meantime. -YAML Framework -============== -In order to utilize these FRE tools, a distributed YAML structure is required. This framework includes a main model yaml, a compile yaml, a platforms yaml, and post-processing yamls. Throughout the compilation and post-processing steps, combined yamls that will be parsed for information are created. Yamls follow a dictionary-like structure with ``[key]: [value]`` fields. +Yaml files +========== + +About Yaml files +---------------- +FRE uses a distributed set of YAML configuration files that together define a model experiment. +Four types of files are required: + +- **Model yaml** — the hub file that defines reusable properties, paths to the other yamls, and + the list of experiments +- **Compile yaml** — source code repositories, component dependencies, and build instructions +- **Platforms yaml** — site- and compiler-specific settings for each target computing environment +- **Post-processing yamls** — postprocess component definitions, settings, and switches + +The compile and platforms yamls are described in detail under `Build a model`_. The +post-processing yamls are described under `Postprocessing`_. + +Yaml schema +----------- +Machine-readable schemas for all FRE YAML files are maintained in the +`NOAA-GFDL/gfdl_msd_schemas `_ +repository. To validate a yaml against the schema locally: + +.. code-block:: console + + fre yamltools validate-yaml -y model.yaml -e experiment_name -p platform -t target + +Variables in FRE Yaml files +---------------------------- +FRE expands several types of variables when processing yaml files. + +*Environment variables* — Standard shell environment variables such as ``$USER``, ``$HOME``, +and ``$ARCHIVE`` are expanded by FRE at runtime. + +*User-defined properties* — The ``fre_properties`` block in the model yaml defines YAML anchors +(``&name value``) that can be referenced elsewhere with ``*name``: + +.. code-block:: console + + fre_properties: + - &group_name "am5" + - &version "2024.01" + +*The* ``!join`` *constructor* — Combines multiple anchors and strings into a single value, +useful for building directory paths: + +.. code-block:: console + + pp_dir: !join [/archive/$USER/, *group_name, /, *version, /, pp] + +Directories +----------- +The ``directories:`` key in the post-processing settings yaml sets the paths for all managed FRE +output: + +.. code-block:: console + + directories: + history_dir: /path/to/raw/model/history/tarballs (required) + pp_dir: /path/to/postprocessed/output (required) + ptmp_dir: /path/to/scratch/working/space (required) + refined_history_dir: /path/to/refinediag/output (required if using refineDiag) + analysis_dir: /path/to/analysis/script/output (required if running analysis scripts) Yaml Formatting --------------- @@ -27,22 +89,22 @@ Post-processing Yamls --------------------- .. include:: usage/yaml_dev/pp_yaml.rst -Build FMS model -=============== +Build a model +============= .. include:: usage/build_fms_model.rst .. include:: usage/guides/fre_make_guide.rst -Run FMS model -============= +Running an experiment +===================== Running experiments via fre-cli is slated for the FRE 2026.03 release. In the meantime, continue to use the legacy FRE Bronx ``frerun`` tool; documentation is available at the `FRE Bronx running an experiment `_ page. -Postprocess FMS history output -============================== +Postprocessing +============== .. include:: usage/postprocess.rst .. include:: usage/guides/fre_pp_guide.rst -Generate data catalogs -====================== -.. include:: tools/catalog.rst +User plugins +============ +.. include:: usage/user_plugins.rst diff --git a/docs/usage/postprocess.rst b/docs/usage/postprocess.rst index f3657e73f..8757f5b8c 100644 --- a/docs/usage/postprocess.rst +++ b/docs/usage/postprocess.rst @@ -1,10 +1,19 @@ -``fre pp`` regrids FMS history files and generates timeseries, climatologies, and static postprocessed files, with instructions specified in YAML. +About postprocessing +-------------------- +``fre pp`` regrids FMS history files and generates timeseries, climatologies, and static +postprocessed files, with instructions specified in YAML. Postprocessing is orchestrated via a +Cylc workflow (fre-workflows) that is configured, installed, and run with the ``fre pp`` +subtools. -User plug-in scripts — refineDiag, preAnalysis, and legacy cshell analysis scripts — can be configured to run before and after postprocessing; see the dedicated sections below. +In the future, output NetCDF files will be rewritten by CMOR by default, ready for publication +to community archives (e.g. ESGF). Presently, standalone CMOR tooling is available as +``fre cmor``. -In the future, output NetCDF files will be rewritten by CMOR by default, ready for publication to community archives (e.g. ESGF). Presently, standalone CMOR tooling is available as ``fre cmor``. +By default, an intake-esm-compatible data catalog is generated and updated, containing a +programmatic metadata-enriched searchable interface to the postprocessed output. The catalog +tooling can be independently assessed as ``fre catalog``. -By default, an intake-esm-compatible data catalog is generated and updated, containing a programmatic metadata-enriched searchable interface to the postprocessed output. The catalog tooling can be independently assessed as ``fre catalog``. +User plug-in scripts can be configured to run at various stages; see the User plugins section. FMS history files ----------------- @@ -115,6 +124,61 @@ Required configuration - history_file: "ocean_month" - history_file: "ocean_annual" +Online postprocessing +--------------------- +In online postprocessing, Bronx ``frerun`` automatically triggers ``fre pp`` after each model +segment completes. This mechanism is provisional and will be improved in a future release. + +**Requirements for online postprocessing:** + +1. The pp yaml must be named identically to the XML file, with only the extension changed + (e.g. ``am5.xml`` → ``am5.yaml``). +2. The yaml must be placed in the FRE ``includeDir`` so that ``frerun`` transfers it to GFDL + along with the runscript. +3. The experiment name in the yaml must match the experiment name in the XML exactly. +4. The ``pp_start`` and ``pp_stop`` years in the yaml must match the run years in the XML. +5. Validate the yaml against the schema on gaea before submitting the run: + +.. code-block:: console + + fre yamltools validate-yaml -y am5.yaml -e experiment_name -p platform -t target + +When these conditions are met, Bronx's ``pp.starter`` runs ``fre pp all`` after each completed +segment. ``fre pp all`` checks whether the postprocessing workflow is already installed and +running — if so, it calls ``fre pp trigger`` to process that segment; if not, it runs the full +setup sequence (``fre pp checkout``, ``configure-yaml``, ``install``, ``run``) before triggering. + +Offline postprocessing +---------------------- +In offline postprocessing, all history files already exist on archive and postprocessing is +run after the fact. This is the typical case when reprocessing an experiment or postprocessing +data from a completed run. + +Set ``history_dir``, ``pp_start``, and ``pp_stop`` to cover the existing history, then run +``fre pp all`` (or ``fre pp run`` followed by manual triggers): + +.. code-block:: console + + fre pp all -e experiment -p platform -T target -c model.yaml + +To process specific time windows, use ``fre pp trigger`` for each segment: + +.. code-block:: console + + fre pp trigger -e experiment -p platform -T target -t 19790101 + fre pp trigger -e experiment -p platform -T target -t 19800101 + +Postprocessing another experiment's output +------------------------------------------- +To postprocess history files from another experiment (including someone else's data), set +``history_dir`` to the path of that experiment's history archive. No other changes are required +— fre pp reads history files from whatever path ``history_dir`` points to. + +.. code-block:: console + + directories: + history_dir: /archive/otherusername/am5/am5f7c1r0/c96L65_am5f7c1r0_amip/gfdl.ncrc5-deploy-prod-openmp/history + XY-regridding ------------- Commonly, native grid history files are regridded during postprocessing. To regrid to a lat/lon grid, configure your @@ -163,6 +227,30 @@ Optional configuration (i.e. if xy-regridding is desired) interpMethod: conserve_order1 +Pressure-level history output +------------------------------ +There are two approaches to obtaining postprocessed atmosphere output on pressure levels. + +**New approach (recommended):** Request pressure-level interpolation directly from the model +by configuring the desired pressure levels in the diag yaml. The model produces history files +already on pressure levels. Because these files contain data at all configured pressure levels +across the full column — including levels below the Earth's surface or above the model top — +those unphysical values must be masked. Enable surface masking with: + +.. code-block:: console + + postprocess: + switches: + do_atmos_plevel_masking: True + +When enabled, atmosphere pressure-level variables are masked below the surface pressure and +above the model top. + +**Old approach (no longer supported):** In FRE Bronx, the model output native vertical levels +and frepp regridded vertically to standard pressure-level sets (ncep, am3, hs20). This +vertical-regrid approach is not available in fre-cli; use the diag yaml interpolation approach +instead. + Timeseries ---------- Timeseries output is the most common type of postprocessed output. Each timeseries file contains @@ -179,6 +267,21 @@ length — how many years of data to combine into one file — is configured glo Timeseries files are organized under ``pp//ts///``. +Sub-year postprocessing +----------------------- +Sub-year chunk lengths are supported using sub-year ISO8601 durations in ``pp_chunks``. For +example, to produce 6-month timeseries files: + +.. code-block:: console + + postprocess: + settings: + pp_chunks: + - P6M + +Any ISO8601 period is valid: ``P1M`` (monthly), ``P3M`` (quarterly), ``P6M`` (semi-annual). +The history segment length must be a multiple of the chunk length. + Climatologies ------------- Climatologies are time-averaged files. All variables from a component appear in one file, with one @@ -206,91 +309,39 @@ Static fields are time-invariant diagnostics such as grid geometry, land/sea mas topography. They are extracted from the nominated source history files and placed in ``pp//ts/static/1yr/``. -Surface masking for FMS pressure-level history ----------------------------------------------- -Pressure-level atmosphere history files contain data at all configured pressure levels, including -levels below the Earth's surface or above the model top. These out-of-range values are unphysical -and should be masked. Enable surface masking with the following switch: +Monthly data from daily data +----------------------------- +This feature is not yet implemented in fre-cli. See the legacy FRE Bronx documentation for the +equivalent ``frepp`` functionality (the ``averageOf`` timeSeries attribute). -.. code-block:: console +Long timeseries from existing timeseries +----------------------------------------- +This feature is not yet implemented in fre-cli. See the legacy FRE Bronx documentation for the +equivalent ``frepp`` functionality (the ``from`` timeSeries attribute). - postprocess: - switches: - do_atmos_plevel_masking: True - -When enabled, atmosphere pressure-level variables are masked below the surface pressure and above -the model top. - -refineDiag scripts ------------------- -refineDiag scripts are user-provided cshell scripts that run on raw history files before -postprocessing and produce new "spoofed" history files. These new files are then postprocessed -alongside native model output, allowing users to define custom derived diagnostics as if they were -native model output. - -Configure refineDiag scripts under ``postprocess.refinediag``: +Checking for missing postprocessed files +----------------------------------------- +To verify that postprocessed timeseries files contain the expected number of time records: .. code-block:: console - directories: - refined_history_dir: /path/to/refined/history/output + fre pp ppval --path /path/to/pp/component/ts/mon/5yr/ - postprocess: - refinediag: - my_script_label: - script: /absolute/path/to/script.csh - inputs: - - atmos_month - outputs: - - atmos_month_refined - do_refinediag: true - -Output file names must differ from any existing history file names. Multiple refineDiag scripts -can be defined, each under its own label. - -preAnalysis scripts -------------------- -preAnalysis scripts are user-provided cshell scripts that run on raw history files before -postprocessing, similar to refineDiag scripts, but do not produce new history files. They are -suited for user-specific tasks such as populating custom databases or producing output outside the -FRE managed directory structure. - -Configure preAnalysis scripts under ``postprocess.preanalysis``: +To validate history files before postprocessing: .. code-block:: console - postprocess: - preanalysis: - my_preanalysis_label: - script: /absolute/path/to/script.csh - inputs: - - atmos_month - do_preanalysis: true - -Legacy cshell analysis scripts -------------------------------- -Legacy cshell analysis scripts are user-provided scripts that run after postprocessing using -postprocessed output to produce figures or other user-specified output. Unlike refineDiag and -preAnalysis scripts, analysis scripts have no downstream dependencies in the FRE postprocessing -workflow. - -Analysis scripts are configured under the top-level ``analysis:`` key, separate from -``postprocess:``: - -.. code-block:: console + fre pp histval --history /path/to/history/ --date_string 19790101 - directories: - analysis_dir: /path/to/analysis/scripts +Filling gaps in postprocessed output +-------------------------------------- +If postprocessing failed or was interrupted for some segments, use ``fre pp trigger`` to +re-trigger processing for specific time windows: - analysis: - my_analysis_label: - legacy: - script: /path/to/analysis_script.csh - required: - data_frequency: mon - date_range: ['1979', '2020'] +.. code-block:: console -Each analysis script is associated with one data frequency. For scripts that span multiple -components, associate the script with the component that has the longest run interval. + fre pp trigger -e experiment -p platform -T target -t 19820101 -A new Python-native analysis framework is being developed for the FRE 2026.02 release. +Common causes of gaps: filesystem failures, history files not yet available on archive, or +interrupted workflow runs. Identify missing segments by comparing the expected date range +against the files present in ``pp_dir``, or use ``fre pp ppval`` to report incomplete files. diff --git a/docs/usage/user_plugins.rst b/docs/usage/user_plugins.rst new file mode 100644 index 000000000..b50a6ac21 --- /dev/null +++ b/docs/usage/user_plugins.rst @@ -0,0 +1,77 @@ +User plug-in scripts run custom code at defined points in the postprocessing workflow. Three +types are supported: refineDiag and preAnalysis scripts run on raw history files before +postprocessing; legacy cshell analysis scripts run after postprocessing on processed output. + +refineDiag scripts +------------------ +refineDiag scripts are user-provided cshell scripts that run on raw history files before +postprocessing and produce new "spoofed" history files. These new files are then postprocessed +alongside native model output, allowing users to define custom derived diagnostics as if they were +native model output. + +Configure refineDiag scripts under ``postprocess.refinediag``: + +.. code-block:: console + + directories: + refined_history_dir: /path/to/refined/history/output + + postprocess: + refinediag: + my_script_label: + script: /absolute/path/to/script.csh + inputs: + - atmos_month + outputs: + - atmos_month_refined + do_refinediag: true + +Output file names must differ from any existing history file names. Multiple refineDiag scripts +can be defined, each under its own label. + +preAnalysis scripts +------------------- +preAnalysis scripts are user-provided cshell scripts that run on raw history files before +postprocessing, similar to refineDiag scripts, but do not produce new history files. They are +suited for user-specific tasks such as populating custom databases or producing output outside the +FRE managed directory structure. + +Configure preAnalysis scripts under ``postprocess.preanalysis``: + +.. code-block:: console + + postprocess: + preanalysis: + my_preanalysis_label: + script: /absolute/path/to/script.csh + inputs: + - atmos_month + do_preanalysis: true + +Legacy cshell analysis scripts +------------------------------- +Legacy cshell analysis scripts are user-provided scripts that run after postprocessing using +postprocessed output to produce figures or other user-specified output. Unlike refineDiag and +preAnalysis scripts, analysis scripts have no downstream dependencies in the FRE postprocessing +workflow. + +Analysis scripts are configured under the top-level ``analysis:`` key, separate from +``postprocess:``: + +.. code-block:: console + + directories: + analysis_dir: /path/to/analysis/scripts + + analysis: + my_analysis_label: + legacy: + script: /path/to/analysis_script.csh + required: + data_frequency: mon + date_range: ['1979', '2020'] + +Each analysis script is associated with one data frequency. For scripts that span multiple +components, associate the script with the component that has the longest run interval. + +A new Python-native analysis framework is being developed for the FRE 2026.02 release. From 57d89b78bce1ae59b5a3dbdf73f74bf8f2b4ce74 Mon Sep 17 00:00:00 2001 From: Chris Blanton Date: Wed, 1 Jul 2026 09:47:15 -0400 Subject: [PATCH 3/5] Restructure Setup page; move dev install to For developers Lead Setup with module load instructions for NOAA-GFDL systems (GFDL one-liner, Gaea with module use first), followed by conda install for generic systems. Remove README pointer and local-clone section from Setup; expand For developers Local/Editable Installation with the actual clone + conda env + pip install steps. Co-Authored-By: Claude Sonnet 4.6 --- docs/for-developers.rst | 26 ++++++++++++++----- docs/setup.rst | 56 +++++++++++++++++++---------------------- 2 files changed, 46 insertions(+), 36 deletions(-) diff --git a/docs/for-developers.rst b/docs/for-developers.rst index c12e9e958..b125660d1 100644 --- a/docs/for-developers.rst +++ b/docs/for-developers.rst @@ -19,13 +19,27 @@ or replace with your fork's link (recommended). Local/Editable Installation --------------------------- -Developers can test local changes by running a ``pip install [-e] .`` inside of the root directory after activating a -virtual environment with ``python>=3.11.*`` and all requirements. This installs the ``fre-cli`` package locally with -any local changes. +Development work on ``fre-cli`` should occur within a conda environment housing ``fre-cli``'s +requirements, with the package installed in editable mode so that local changes take effect +without reinstalling:: -Development work on ``fre-cli`` should occur within a conda environment housing ``fre-cli``'s requirements, and -a local copy of the repository to install with ``pip`` using the ``-e/--editable`` flag on. This specific approach is -described in :ref:`setup`. + # fre-cli needs only these two channels + conda config --append channels noaa-gfdl + conda config --append channels conda-forge + + # clone the repository (--recursive pulls in submodules such as mkmf) + git clone --recursive https://github.com/noaa-gfdl/fre-cli.git + cd fre-cli + + # create the conda environment with all dependencies (fre-cli itself is not installed yet) + conda env create -f environment.yml + conda activate fre-cli + + # add mkmf to PATH + export PATH=$PATH:${PWD}/mkmf/bin + + # install fre-cli in editable mode — no reinstall needed after editing source files + pip install -e . Testing Your Local Changes and Installation diff --git a/docs/setup.rst b/docs/setup.rst index 1556c9c28..7b1a1729a 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -3,48 +3,44 @@ ===== Setup ===== -FRE-cli is conda-installable from the “noaa-gfdl” anaconda channel (https://anaconda.org/NOAA-GFDL/fre-cli) -and is deployed on NOAA RDHPCS and GFDL systems as Environment Modules. +FRE-cli is deployed on NOAA RDHPCS and GFDL systems as Environment Modules and is +conda-installable from the ``noaa-gfdl`` anaconda channel for use on any system. -Installing FRE-cli -================== +Loading FRE-cli on NOAA-GFDL systems +===================================== -Instructions for installing fre-cli on GFDL and NOAA RDHPCS systems via lmod and on any system via conda can be found in the `fre-cli repository README `__. +The GFDL Workflow Team deploys fre-cli as an Environment Module on GFDL workstations, PPAN, +and Gaea. +At GFDL (workstations and PPAN): -Install FRE-cli from local GitHub clone (for development) -========================================================= +.. code-block:: console -When contributing to FRE, it is recommended to download and use the code from the repository directly to test changes. -Assuming one has ``conda`` in their path, then do the following:: + module load fre/ - # Append necessary channels- fre-cli needs only these two channels and no others to build. - # it's possible depending on your conda installation that additional configuration steps are needed - conda config --append channels noaa-gfdl - conda config --append channels conda-forge +On Gaea, the FRE modulefiles must be added to the module search path first: - # grab a copy of the code from github and cd into the repository directory - git clone --recursive https://github.com/noaa-gfdl/fre-cli.git - cd fre-cli +.. code-block:: console - # to avoid being prompted for confirmation, add '-y' to the call - # this downloads/builds fre-cli's dependencies ONLY - conda env create -f environment.yml + module use -a /ncrc/home2/fms/local/modulefiles + module load fre/ - # activate the environment you just created. - # fre-cli isn't installed yet though, ONLY dependencies - # if you changed the name of the build environment, activate that name instead of fre-cli - conda activate fre-cli +Replace ```` with the desired release (e.g. ``2026.01``). - # add mkmf to your PATH - export PATH=$PATH:${PWD}/mkmf/bin +Installing FRE-cli on generic systems +====================================== - # now we pip install the local code under the `fre/` directory - # the -e flag makes re-installing the code after editing not necessary - pip install -e . +On any system with conda available, fre-cli can be installed from the ``noaa-gfdl`` anaconda +channel: - # optional: load fre-nctools into your PATH to gain access to regridding and certain time-averaging routines - # module load fre-nctools +.. code-block:: console + + conda config --append channels noaa-gfdl + conda config --append channels conda-forge + conda create --name fre- --channel noaa-gfdl --channel conda-forge fre-cli:: + conda activate fre- + +Replace ```` with the desired release (e.g. ``2026.01``). Using the container =================== From 4ef3c8e7d7236c2fe266f5cb6e8927509e9b8065 Mon Sep 17 00:00:00 2001 From: Chris Blanton Date: Wed, 1 Jul 2026 09:53:12 -0400 Subject: [PATCH 4/5] Add Configuration examples and AI assistance sections New top-level pages linking to public fre-examples repo and internal GFDL MSD configurations, plus a stub AI assistance page for the forthcoming chatbot offering. Co-Authored-By: Claude Sonnet 4.6 --- docs/ai-assistance.rst | 9 +++++++++ docs/examples.rst | 15 +++++++++++++++ docs/index.rst | 2 ++ 3 files changed, 26 insertions(+) create mode 100644 docs/ai-assistance.rst create mode 100644 docs/examples.rst diff --git a/docs/ai-assistance.rst b/docs/ai-assistance.rst new file mode 100644 index 000000000..6055a0cd5 --- /dev/null +++ b/docs/ai-assistance.rst @@ -0,0 +1,9 @@ +.. _ai-assistance: + +============= +AI assistance +============= + +A chatbot is available to help with all aspects of FRE configuration, setup, and debugging. + +.. TODO: add chatbot URL and access instructions diff --git a/docs/examples.rst b/docs/examples.rst new file mode 100644 index 000000000..36e3bc0c0 --- /dev/null +++ b/docs/examples.rst @@ -0,0 +1,15 @@ +===================== +Configuration examples +===================== + +Public configurations +===================== +The `fre-examples `_ repository contains publicly +available example FRE YAML configuration files. These configurations demonstrate common +experiment setups and serve as starting points for new experiments. + +GFDL Modeling Systems Division configurations +============================================= +Supported FRE configurations maintained by the GFDL Modeling Systems Division are available at +https://gitlab.gfdl.noaa.gov/fms/xml (GFDL internal access required). These include production +configurations for GFDL climate and weather models. diff --git a/docs/index.rst b/docs/index.rst index 4d9259418..3de31d50c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,6 +18,8 @@ Welcome to ``fre-cli``'s documentation! setup tools usage + examples + ai-assistance api for-developers glossary From be7bd5d73ccb0cc9c158548e1979521e46f1683f Mon Sep 17 00:00:00 2001 From: Chris Blanton Date: Wed, 1 Jul 2026 17:03:02 -0400 Subject: [PATCH 5/5] Remove AI assistance page (not yet ready) Co-Authored-By: Claude Sonnet 4.6 --- docs/ai-assistance.rst | 9 --------- docs/index.rst | 1 - 2 files changed, 10 deletions(-) delete mode 100644 docs/ai-assistance.rst diff --git a/docs/ai-assistance.rst b/docs/ai-assistance.rst deleted file mode 100644 index 6055a0cd5..000000000 --- a/docs/ai-assistance.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _ai-assistance: - -============= -AI assistance -============= - -A chatbot is available to help with all aspects of FRE configuration, setup, and debugging. - -.. TODO: add chatbot URL and access instructions diff --git a/docs/index.rst b/docs/index.rst index 3de31d50c..a5b104921 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -19,7 +19,6 @@ Welcome to ``fre-cli``'s documentation! tools usage examples - ai-assistance api for-developers glossary