diff --git a/docs/container.rst b/docs/container.rst index aae515545..e3e0446c2 100644 --- a/docs/container.rst +++ b/docs/container.rst @@ -1,51 +1,9 @@ Post-Processing Container ========================= -Previously, many GFDL workflows and configurations have only been accessible on gitlab. This is disadvantageous for outside collaboration, flexibility, community development. While the FRE workflow can now be conda installed, another deployment method of containerization has been developed. Containerzation of the FRE workflow at GFDL bolsters portability while also simplifying the environment set-up for the user. With the environment set-up done through the container build and runscript, this post-processing container work allows for more effective sharing of the workflow. - -BUILDING --------- - -In order to build the container, the user needs to have podman access on gaea. If needed, put in a helpdesk ticket. - -Files used to build container: - - Dockerfile-ppp - - cylc-flow-tools environment yaml - - runscript.sh - -The container will house the fre-cli tools and subtools, and any necessary packages needed for those tools. - -Using podman and apptainer to build, follow these steps: - -1. Clone the HPC-ME repository - -.. code-block:: console - - git clone git@gitlab.gfdl.noaa.gov:fre/HPC-ME.git - -2. Navigate into the ppp folder - -.. code-block:: console - - cd HPC-ME/ppp - -3. Build a container image - -.. code-block:: console - - podman build -f Dockerfile-ppp -t 2025 - -4. Save the image to a local tar file (It is recommended to name the container after the post-processing experiment name) - -.. code-block:: console - - podman save -o [name of container].tar localhost/2025 - -5. Create the singularity image file (sif) from the tar file - -.. code-block:: console - - apptainer build --disable-cache [name of container].sif docker-archive://[name of container].tar +The FRE post-processing container packages fre-cli and its dependencies into a portable +Apptainer/Singularity image, enabling postprocessing on systems where a full conda install is +not practical. The container is distributed as a pre-built ``.sif`` file. SETUP ----- diff --git a/docs/examples.rst b/docs/examples.rst new file mode 100644 index 000000000..36e3bc0c0 --- /dev/null +++ b/docs/examples.rst @@ -0,0 +1,15 @@ +===================== +Configuration examples +===================== + +Public configurations +===================== +The `fre-examples `_ repository contains publicly +available example FRE YAML configuration files. These configurations demonstrate common +experiment setups and serve as starting points for new experiments. + +GFDL Modeling Systems Division configurations +============================================= +Supported FRE configurations maintained by the GFDL Modeling Systems Division are available at +https://gitlab.gfdl.noaa.gov/fms/xml (GFDL internal access required). These include production +configurations for GFDL climate and weather models. diff --git a/docs/for-developers.rst b/docs/for-developers.rst index c12e9e958..b125660d1 100644 --- a/docs/for-developers.rst +++ b/docs/for-developers.rst @@ -19,13 +19,27 @@ or replace with your fork's link (recommended). Local/Editable Installation --------------------------- -Developers can test local changes by running a ``pip install [-e] .`` inside of the root directory after activating a -virtual environment with ``python>=3.11.*`` and all requirements. This installs the ``fre-cli`` package locally with -any local changes. +Development work on ``fre-cli`` should occur within a conda environment housing ``fre-cli``'s +requirements, with the package installed in editable mode so that local changes take effect +without reinstalling:: -Development work on ``fre-cli`` should occur within a conda environment housing ``fre-cli``'s requirements, and -a local copy of the repository to install with ``pip`` using the ``-e/--editable`` flag on. This specific approach is -described in :ref:`setup`. + # fre-cli needs only these two channels + conda config --append channels noaa-gfdl + conda config --append channels conda-forge + + # clone the repository (--recursive pulls in submodules such as mkmf) + git clone --recursive https://github.com/noaa-gfdl/fre-cli.git + cd fre-cli + + # create the conda environment with all dependencies (fre-cli itself is not installed yet) + conda env create -f environment.yml + conda activate fre-cli + + # add mkmf to PATH + export PATH=$PATH:${PWD}/mkmf/bin + + # install fre-cli in editable mode — no reinstall needed after editing source files + pip install -e . Testing Your Local Changes and Installation diff --git a/docs/index.rst b/docs/index.rst index be868d80b..a5b104921 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,9 +16,9 @@ Welcome to ``fre-cli``'s documentation! what-is-fre setup - usage tools - container + usage + examples api for-developers glossary diff --git a/docs/setup.rst b/docs/setup.rst index 55746b776..7b1a1729a 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -3,45 +3,46 @@ ===== Setup ===== -FRE-cli is conda-installable from the “noaa-gfdl” anaconda channel (https://anaconda.org/NOAA-GFDL/fre-cli) -and is deployed on NOAA RDHPCS and GFDL systems as Environment Modules. +FRE-cli is deployed on NOAA RDHPCS and GFDL systems as Environment Modules and is +conda-installable from the ``noaa-gfdl`` anaconda channel for use on any system. -Installing FRE-cli -================== +Loading FRE-cli on NOAA-GFDL systems +===================================== -Instructions for installing fre-cli on GFDL and NOAA RDHPCS systems via lmod and on any system via conda can be found in the `fre-cli repository README `__. +The GFDL Workflow Team deploys fre-cli as an Environment Module on GFDL workstations, PPAN, +and Gaea. +At GFDL (workstations and PPAN): -Install FRE-cli from local GitHub clone (for development) -========================================================= +.. code-block:: console -When contributing to FRE, it is recommended to download and use the code from the repository directly to test changes. -Assuming one has ``conda`` in their path, then do the following:: + module load fre/ - # Append necessary channels- fre-cli needs only these two channels and no others to build. - # it's possible depending on your conda installation that additional configuration steps are needed - conda config --append channels noaa-gfdl - conda config --append channels conda-forge +On Gaea, the FRE modulefiles must be added to the module search path first: - # grab a copy of the code from github and cd into the repository directory - git clone --recursive https://github.com/noaa-gfdl/fre-cli.git - cd fre-cli +.. code-block:: console - # to avoid being prompted for confirmation, add '-y' to the call - # this downloads/builds fre-cli's dependencies ONLY - conda env create -f environment.yml + module use -a /ncrc/home2/fms/local/modulefiles + module load fre/ - # activate the environment you just created. - # fre-cli isn't installed yet though, ONLY dependencies - # if you changed the name of the build environment, activate that name instead of fre-cli - conda activate fre-cli +Replace ```` with the desired release (e.g. ``2026.01``). - # add mkmf to your PATH - export PATH=$PATH:${PWD}/mkmf/bin +Installing FRE-cli on generic systems +====================================== - # now we pip install the local code under the `fre/` directory - # the -e flag makes re-installing the code after editing not necessary - pip install -e . +On any system with conda available, fre-cli can be installed from the ``noaa-gfdl`` anaconda +channel: - # optional: load fre-nctools into your PATH to gain access to regridding and certain time-averaging routines - # module load fre-nctools +.. code-block:: console + + conda config --append channels noaa-gfdl + conda config --append channels conda-forge + conda create --name fre- --channel noaa-gfdl --channel conda-forge fre-cli:: + conda activate fre- + +Replace ```` with the desired release (e.g. ``2026.01``). + +Using the container +=================== + +.. include:: container.rst diff --git a/docs/tools.rst b/docs/tools.rst index 655028ee8..c39e27265 100644 --- a/docs/tools.rst +++ b/docs/tools.rst @@ -1,8 +1,8 @@ .. last updated May 2025 -===== -Tools -===== +========= +FRE tools +========= ``fre`` ======= diff --git a/docs/tools/pp.rst b/docs/tools/pp.rst index 8190102b7..15654c053 100644 --- a/docs/tools/pp.rst +++ b/docs/tools/pp.rst @@ -1,6 +1,4 @@ -.. NEEDS UPDATING #TODO - - +For conceptual background and step-by-step workflow instructions, see the "Postprocess FMS history output" section in the Usage guide. ``all`` ------- diff --git a/docs/usage.rst b/docs/usage.rst index d6064b6aa..b0b757fcc 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1,11 +1,73 @@ ===== Usage ===== -Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Please note that model running is not yet supported in FRE 2025; continue to use FRE Bronx frerun. +Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model and ``fre pp`` +postprocesses the history output and runs diagnostic analysis scripts. Model running is slated for +the FRE 2026.03 release; continue to use FRE Bronx frerun in the meantime. -YAML Framework -============== -In order to utilize these FRE tools, a distributed YAML structure is required. This framework includes a main model yaml, a compile yaml, a platforms yaml, and post-processing yamls. Throughout the compilation and post-processing steps, combined yamls that will be parsed for information are created. Yamls follow a dictionary-like structure with ``[key]: [value]`` fields. +Yaml files +========== + +About Yaml files +---------------- +FRE uses a distributed set of YAML configuration files that together define a model experiment. +Four types of files are required: + +- **Model yaml** — the hub file that defines reusable properties, paths to the other yamls, and + the list of experiments +- **Compile yaml** — source code repositories, component dependencies, and build instructions +- **Platforms yaml** — site- and compiler-specific settings for each target computing environment +- **Post-processing yamls** — postprocess component definitions, settings, and switches + +The compile and platforms yamls are described in detail under `Build a model`_. The +post-processing yamls are described under `Postprocessing`_. + +Yaml schema +----------- +Machine-readable schemas for all FRE YAML files are maintained in the +`NOAA-GFDL/gfdl_msd_schemas `_ +repository. To validate a yaml against the schema locally: + +.. code-block:: console + + fre yamltools validate-yaml -y model.yaml -e experiment_name -p platform -t target + +Variables in FRE Yaml files +---------------------------- +FRE expands several types of variables when processing yaml files. + +*Environment variables* — Standard shell environment variables such as ``$USER``, ``$HOME``, +and ``$ARCHIVE`` are expanded by FRE at runtime. + +*User-defined properties* — The ``fre_properties`` block in the model yaml defines YAML anchors +(``&name value``) that can be referenced elsewhere with ``*name``: + +.. code-block:: console + + fre_properties: + - &group_name "am5" + - &version "2024.01" + +*The* ``!join`` *constructor* — Combines multiple anchors and strings into a single value, +useful for building directory paths: + +.. code-block:: console + + pp_dir: !join [/archive/$USER/, *group_name, /, *version, /, pp] + +Directories +----------- +The ``directories:`` key in the post-processing settings yaml sets the paths for all managed FRE +output: + +.. code-block:: console + + directories: + history_dir: /path/to/raw/model/history/tarballs (required) + pp_dir: /path/to/postprocessed/output (required) + ptmp_dir: /path/to/scratch/working/space (required) + refined_history_dir: /path/to/refinediag/output (required if using refineDiag) + analysis_dir: /path/to/analysis/script/output (required if running analysis scripts) Yaml Formatting --------------- @@ -27,20 +89,22 @@ Post-processing Yamls --------------------- .. include:: usage/yaml_dev/pp_yaml.rst -Build FMS model -=============== +Build a model +============= .. include:: usage/build_fms_model.rst .. include:: usage/guides/fre_make_guide.rst -Run FMS model -============= -Check back in the latter half of 2025 or so. +Running an experiment +===================== +Running experiments via fre-cli is slated for the FRE 2026.03 release. In the meantime, continue +to use the legacy FRE Bronx ``frerun`` tool; documentation is available at the +`FRE Bronx running an experiment `_ page. -Postprocess FMS history output -============================== +Postprocessing +============== .. include:: usage/postprocess.rst .. include:: usage/guides/fre_pp_guide.rst -Generate data catalogs -====================== -.. include:: tools/catalog.rst +User plugins +============ +.. include:: usage/user_plugins.rst diff --git a/docs/usage/postprocess.rst b/docs/usage/postprocess.rst index 60d3cc651..8757f5b8c 100644 --- a/docs/usage/postprocess.rst +++ b/docs/usage/postprocess.rst @@ -1,10 +1,19 @@ -``fre pp`` regrids FMS history files and generates timeseries, climatologies, and static postprocessed files, with instructions specified in YAML. +About postprocessing +-------------------- +``fre pp`` regrids FMS history files and generates timeseries, climatologies, and static +postprocessed files, with instructions specified in YAML. Postprocessing is orchestrated via a +Cylc workflow (fre-workflows) that is configured, installed, and run with the ``fre pp`` +subtools. -Bronx plug-in refineDiag and analysis scripts can also be used, and a reimagined analysis script ecosystem is being developed and is available now (for adventurous users). The new analysis script framework is independent of and compatible with FRE (https://github.com/NOAA-GFDL/analysis-scripts). The goal is to combine the ease-of-use of legacy FRE analysis scripts with the standardization of model output data catalogs and python virtual environments. +In the future, output NetCDF files will be rewritten by CMOR by default, ready for publication +to community archives (e.g. ESGF). Presently, standalone CMOR tooling is available as +``fre cmor``. -In the future, output NetCDF files will be rewritten by CMOR by default, ready for publication to community archives (e.g. ESGF). Presently, standalone CMOR tooling is available as ``fre cmor``. +By default, an intake-esm-compatible data catalog is generated and updated, containing a +programmatic metadata-enriched searchable interface to the postprocessed output. The catalog +tooling can be independently assessed as ``fre catalog``. -By default, an intake-esm-compatible data catalog is generated and updated, containing a programmatic metadata-enriched searchable interface to the postprocessed output. The catalog tooling can be independently assessed as ``fre catalog``. +User plug-in scripts can be configured to run at various stages; see the User plugins section. FMS history files ----------------- @@ -115,6 +124,61 @@ Required configuration - history_file: "ocean_month" - history_file: "ocean_annual" +Online postprocessing +--------------------- +In online postprocessing, Bronx ``frerun`` automatically triggers ``fre pp`` after each model +segment completes. This mechanism is provisional and will be improved in a future release. + +**Requirements for online postprocessing:** + +1. The pp yaml must be named identically to the XML file, with only the extension changed + (e.g. ``am5.xml`` → ``am5.yaml``). +2. The yaml must be placed in the FRE ``includeDir`` so that ``frerun`` transfers it to GFDL + along with the runscript. +3. The experiment name in the yaml must match the experiment name in the XML exactly. +4. The ``pp_start`` and ``pp_stop`` years in the yaml must match the run years in the XML. +5. Validate the yaml against the schema on gaea before submitting the run: + +.. code-block:: console + + fre yamltools validate-yaml -y am5.yaml -e experiment_name -p platform -t target + +When these conditions are met, Bronx's ``pp.starter`` runs ``fre pp all`` after each completed +segment. ``fre pp all`` checks whether the postprocessing workflow is already installed and +running — if so, it calls ``fre pp trigger`` to process that segment; if not, it runs the full +setup sequence (``fre pp checkout``, ``configure-yaml``, ``install``, ``run``) before triggering. + +Offline postprocessing +---------------------- +In offline postprocessing, all history files already exist on archive and postprocessing is +run after the fact. This is the typical case when reprocessing an experiment or postprocessing +data from a completed run. + +Set ``history_dir``, ``pp_start``, and ``pp_stop`` to cover the existing history, then run +``fre pp all`` (or ``fre pp run`` followed by manual triggers): + +.. code-block:: console + + fre pp all -e experiment -p platform -T target -c model.yaml + +To process specific time windows, use ``fre pp trigger`` for each segment: + +.. code-block:: console + + fre pp trigger -e experiment -p platform -T target -t 19790101 + fre pp trigger -e experiment -p platform -T target -t 19800101 + +Postprocessing another experiment's output +------------------------------------------- +To postprocess history files from another experiment (including someone else's data), set +``history_dir`` to the path of that experiment's history archive. No other changes are required +— fre pp reads history files from whatever path ``history_dir`` points to. + +.. code-block:: console + + directories: + history_dir: /archive/otherusername/am5/am5f7c1r0/c96L65_am5f7c1r0_amip/gfdl.ncrc5-deploy-prod-openmp/history + XY-regridding ------------- Commonly, native grid history files are regridded during postprocessing. To regrid to a lat/lon grid, configure your @@ -163,27 +227,121 @@ Optional configuration (i.e. if xy-regridding is desired) interpMethod: conserve_order1 +Pressure-level history output +------------------------------ +There are two approaches to obtaining postprocessed atmosphere output on pressure levels. + +**New approach (recommended):** Request pressure-level interpolation directly from the model +by configuring the desired pressure levels in the diag yaml. The model produces history files +already on pressure levels. Because these files contain data at all configured pressure levels +across the full column — including levels below the Earth's surface or above the model top — +those unphysical values must be masked. Enable surface masking with: + +.. code-block:: console + + postprocess: + switches: + do_atmos_plevel_masking: True + +When enabled, atmosphere pressure-level variables are masked below the surface pressure and +above the model top. + +**Old approach (no longer supported):** In FRE Bronx, the model output native vertical levels +and frepp regridded vertically to standard pressure-level sets (ncep, am3, hs20). This +vertical-regrid approach is not available in fre-cli; use the diag yaml interpolation approach +instead. + Timeseries ---------- -Timeseries output is the most common type of postprocessed output. +Timeseries output is the most common type of postprocessed output. Each timeseries file contains +one variable from one postprocess component, spanning the requested date range. The temporal chunk +length — how many years of data to combine into one file — is configured globally in +``postprocess.settings.pp_chunks`` as an ISO8601 duration: + +.. code-block:: console + + postprocess: + settings: + pp_chunks: + - P5Y + +Timeseries files are organized under ``pp//ts///``. + +Sub-year postprocessing +----------------------- +Sub-year chunk lengths are supported using sub-year ISO8601 durations in ``pp_chunks``. For +example, to produce 6-month timeseries files: + +.. code-block:: console + + postprocess: + settings: + pp_chunks: + - P6M + +Any ISO8601 period is valid: ``P1M`` (monthly), ``P3M`` (quarterly), ``P6M`` (semi-annual). +The history segment length must be a multiple of the chunk length. Climatologies ------------- -annual and monthly climatologies -less fine-grained than bronx -per-component switch coming -now it's one switch for entire pp +Climatologies are time-averaged files. All variables from a component appear in one file, with one +time-averaged level per file. Climatology output is configured per-component under a +``climatology:`` list, with ``frequency`` (``yr`` for annual, ``mon`` for monthly) and +``interval_years`` specifying the averaging period length: + +.. code-block:: console + + postprocess: + components: + - type: atmos_month + climatology: + - frequency: yr + interval_years: 2 + - frequency: mon + interval_years: 2 + +Annual climatologies produce one annually-averaged file per interval. Monthly climatologies produce +twelve monthly-mean files per interval. Statics ------- -underbaked, known deficiency -currently, takes statics from "source" history files +Static fields are time-invariant diagnostics such as grid geometry, land/sea masks, and +topography. They are extracted from the nominated source history files and placed in +``pp//ts/static/1yr/``. + +Monthly data from daily data +----------------------------- +This feature is not yet implemented in fre-cli. See the legacy FRE Bronx documentation for the +equivalent ``frepp`` functionality (the ``averageOf`` timeSeries attribute). + +Long timeseries from existing timeseries +----------------------------------------- +This feature is not yet implemented in fre-cli. See the legacy FRE Bronx documentation for the +equivalent ``frepp`` functionality (the ``from`` timeSeries attribute). + +Checking for missing postprocessed files +----------------------------------------- +To verify that postprocessed timeseries files contain the expected number of time records: + +.. code-block:: console -Analysis scripts ----------------- + fre pp ppval --path /path/to/pp/component/ts/mon/5yr/ + +To validate history files before postprocessing: + +.. code-block:: console + + fre pp histval --history /path/to/history/ --date_string 19790101 + +Filling gaps in postprocessed output +-------------------------------------- +If postprocessing failed or was interrupted for some segments, use ``fre pp trigger`` to +re-trigger processing for specific time windows: + +.. code-block:: console -Surface masking for FMS pressure-level history ----------------------------------------------- + fre pp trigger -e experiment -p platform -T target -t 19820101 -Legacy refineDiag scripts -------------------------- +Common causes of gaps: filesystem failures, history files not yet available on archive, or +interrupted workflow runs. Identify missing segments by comparing the expected date range +against the files present in ``pp_dir``, or use ``fre pp ppval`` to report incomplete files. diff --git a/docs/usage/user_plugins.rst b/docs/usage/user_plugins.rst new file mode 100644 index 000000000..b50a6ac21 --- /dev/null +++ b/docs/usage/user_plugins.rst @@ -0,0 +1,77 @@ +User plug-in scripts run custom code at defined points in the postprocessing workflow. Three +types are supported: refineDiag and preAnalysis scripts run on raw history files before +postprocessing; legacy cshell analysis scripts run after postprocessing on processed output. + +refineDiag scripts +------------------ +refineDiag scripts are user-provided cshell scripts that run on raw history files before +postprocessing and produce new "spoofed" history files. These new files are then postprocessed +alongside native model output, allowing users to define custom derived diagnostics as if they were +native model output. + +Configure refineDiag scripts under ``postprocess.refinediag``: + +.. code-block:: console + + directories: + refined_history_dir: /path/to/refined/history/output + + postprocess: + refinediag: + my_script_label: + script: /absolute/path/to/script.csh + inputs: + - atmos_month + outputs: + - atmos_month_refined + do_refinediag: true + +Output file names must differ from any existing history file names. Multiple refineDiag scripts +can be defined, each under its own label. + +preAnalysis scripts +------------------- +preAnalysis scripts are user-provided cshell scripts that run on raw history files before +postprocessing, similar to refineDiag scripts, but do not produce new history files. They are +suited for user-specific tasks such as populating custom databases or producing output outside the +FRE managed directory structure. + +Configure preAnalysis scripts under ``postprocess.preanalysis``: + +.. code-block:: console + + postprocess: + preanalysis: + my_preanalysis_label: + script: /absolute/path/to/script.csh + inputs: + - atmos_month + do_preanalysis: true + +Legacy cshell analysis scripts +------------------------------- +Legacy cshell analysis scripts are user-provided scripts that run after postprocessing using +postprocessed output to produce figures or other user-specified output. Unlike refineDiag and +preAnalysis scripts, analysis scripts have no downstream dependencies in the FRE postprocessing +workflow. + +Analysis scripts are configured under the top-level ``analysis:`` key, separate from +``postprocess:``: + +.. code-block:: console + + directories: + analysis_dir: /path/to/analysis/scripts + + analysis: + my_analysis_label: + legacy: + script: /path/to/analysis_script.csh + required: + data_frequency: mon + date_range: ['1979', '2020'] + +Each analysis script is associated with one data frequency. For scripts that span multiple +components, associate the script with the component that has the longest run interval. + +A new Python-native analysis framework is being developed for the FRE 2026.02 release.