Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 3 additions & 45 deletions docs/container.rst
Original file line number Diff line number Diff line change
@@ -1,51 +1,9 @@
Post-Processing Container
=========================

Previously, many GFDL workflows and configurations have only been accessible on gitlab. This is disadvantageous for outside collaboration, flexibility, community development. While the FRE workflow can now be conda installed, another deployment method of containerization has been developed. Containerzation of the FRE workflow at GFDL bolsters portability while also simplifying the environment set-up for the user. With the environment set-up done through the container build and runscript, this post-processing container work allows for more effective sharing of the workflow.

BUILDING
--------

In order to build the container, the user needs to have podman access on gaea. If needed, put in a helpdesk ticket.

Files used to build container:
- Dockerfile-ppp
- cylc-flow-tools environment yaml
- runscript.sh

The container will house the fre-cli tools and subtools, and any necessary packages needed for those tools.

Using podman and apptainer to build, follow these steps:

1. Clone the HPC-ME repository

.. code-block:: console

git clone git@gitlab.gfdl.noaa.gov:fre/HPC-ME.git

2. Navigate into the ppp folder

.. code-block:: console

cd HPC-ME/ppp

3. Build a container image

.. code-block:: console

podman build -f Dockerfile-ppp -t 2025

4. Save the image to a local tar file (It is recommended to name the container after the post-processing experiment name)

.. code-block:: console

podman save -o [name of container].tar localhost/2025

5. Create the singularity image file (sif) from the tar file

.. code-block:: console

apptainer build --disable-cache [name of container].sif docker-archive://[name of container].tar
The FRE post-processing container packages fre-cli and its dependencies into a portable
Apptainer/Singularity image, enabling postprocessing on systems where a full conda install is
not practical. The container is distributed as a pre-built ``.sif`` file.

SETUP
-----
Expand Down
15 changes: 15 additions & 0 deletions docs/examples.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
=====================

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check the build logs of your read the docs build.

I see one warning that can easily be resolved
/home/docs/checkouts/readthedocs.org/user_builds/noaa-gfdl-fre-cli/checkouts/925/docs/examples.rst:1: WARNING: Title overline too short.

Configuration examples
=====================

Public configurations
=====================
The `fre-examples <https://github.com/noaa-gfdl/fre-examples>`_ repository contains publicly
available example FRE YAML configuration files. These configurations demonstrate common
experiment setups and serve as starting points for new experiments.

GFDL Modeling Systems Division configurations
=============================================
Supported FRE configurations maintained by the GFDL Modeling Systems Division are available at
https://gitlab.gfdl.noaa.gov/fms/xml (GFDL internal access required). These include production
configurations for GFDL climate and weather models.
26 changes: 20 additions & 6 deletions docs/for-developers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,27 @@ or replace with your fork's link (recommended).

Local/Editable Installation
---------------------------
Developers can test local changes by running a ``pip install [-e] .`` inside of the root directory after activating a
virtual environment with ``python>=3.11.*`` and all requirements. This installs the ``fre-cli`` package locally with
any local changes.
Development work on ``fre-cli`` should occur within a conda environment housing ``fre-cli``'s
requirements, with the package installed in editable mode so that local changes take effect
without reinstalling::

Development work on ``fre-cli`` should occur within a conda environment housing ``fre-cli``'s requirements, and
a local copy of the repository to install with ``pip`` using the ``-e/--editable`` flag on. This specific approach is
described in :ref:`setup`.
# fre-cli needs only these two channels
conda config --append channels noaa-gfdl
conda config --append channels conda-forge

# clone the repository (--recursive pulls in submodules such as mkmf)
git clone --recursive https://github.com/noaa-gfdl/fre-cli.git
cd fre-cli

# create the conda environment with all dependencies (fre-cli itself is not installed yet)
conda env create -f environment.yml
conda activate fre-cli

# add mkmf to PATH
export PATH=$PATH:${PWD}/mkmf/bin

# install fre-cli in editable mode — no reinstall needed after editing source files
pip install -e .


Testing Your Local Changes and Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ Welcome to ``fre-cli``'s documentation!

what-is-fre
setup
usage
tools
container
usage
examples
api
for-developers
glossary
Expand Down
61 changes: 31 additions & 30 deletions docs/setup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,45 +3,46 @@
=====
Setup
=====
FRE-cli is conda-installable from the “noaa-gfdl” anaconda channel (https://anaconda.org/NOAA-GFDL/fre-cli)
and is deployed on NOAA RDHPCS and GFDL systems as Environment Modules.
FRE-cli is deployed on NOAA RDHPCS and GFDL systems as Environment Modules and is
conda-installable from the ``noaa-gfdl`` anaconda channel for use on any system.

Installing FRE-cli
==================
Loading FRE-cli on NOAA-GFDL systems
=====================================

Instructions for installing fre-cli on GFDL and NOAA RDHPCS systems via lmod and on any system via conda can be found in the `fre-cli repository README <https://github.com/NOAA-GFDL/fre-cli/blob/main/README.md>`__.
The GFDL Workflow Team deploys fre-cli as an Environment Module on GFDL workstations, PPAN,
and Gaea.

At GFDL (workstations and PPAN):

Install FRE-cli from local GitHub clone (for development)
=========================================================
.. code-block:: console

When contributing to FRE, it is recommended to download and use the code from the repository directly to test changes.
Assuming one has ``conda`` in their path, then do the following::
module load fre/<version>

# Append necessary channels- fre-cli needs only these two channels and no others to build.
# it's possible depending on your conda installation that additional configuration steps are needed
conda config --append channels noaa-gfdl
conda config --append channels conda-forge
On Gaea, the FRE modulefiles must be added to the module search path first:

# grab a copy of the code from github and cd into the repository directory
git clone --recursive https://github.com/noaa-gfdl/fre-cli.git
cd fre-cli
.. code-block:: console

# to avoid being prompted for confirmation, add '-y' to the call
# this downloads/builds fre-cli's dependencies ONLY
conda env create -f environment.yml
module use -a /ncrc/home2/fms/local/modulefiles
module load fre/<version>

# activate the environment you just created.
# fre-cli isn't installed yet though, ONLY dependencies
# if you changed the name of the build environment, activate that name instead of fre-cli
conda activate fre-cli
Replace ``<version>`` with the desired release (e.g. ``2026.01``).

# add mkmf to your PATH
export PATH=$PATH:${PWD}/mkmf/bin
Installing FRE-cli on generic systems
======================================

# now we pip install the local code under the `fre/` directory
# the -e flag makes re-installing the code after editing not necessary
pip install -e .
On any system with conda available, fre-cli can be installed from the ``noaa-gfdl`` anaconda
channel:

# optional: load fre-nctools into your PATH to gain access to regridding and certain time-averaging routines
# module load fre-nctools
.. code-block:: console

conda config --append channels noaa-gfdl
conda config --append channels conda-forge
conda create --name fre-<version> --channel noaa-gfdl --channel conda-forge fre-cli::<version>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be fre-cli==<version> at the end? Ryan tried this out and conda was reading fre-cli as a channel and not a package

@ceblanton ceblanton Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right you are! (do AI hallucination emojis exist yet?)

conda activate fre-<version>

Replace ``<version>`` with the desired release (e.g. ``2026.01``).

Using the container
===================

.. include:: container.rst
6 changes: 3 additions & 3 deletions docs/tools.rst
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
.. last updated May 2025

=====
Tools
=====
=========
FRE tools
=========

``fre``
=======
Expand Down
4 changes: 1 addition & 3 deletions docs/tools/pp.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
.. NEEDS UPDATING #TODO


For conceptual background and step-by-step workflow instructions, see the "Postprocess FMS history output" section in the Usage guide.

``all``
-------
Expand Down
92 changes: 78 additions & 14 deletions docs/usage.rst
Original file line number Diff line number Diff line change
@@ -1,11 +1,73 @@
=====
Usage
=====
Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Please note that model running is not yet supported in FRE 2025; continue to use FRE Bronx frerun.
Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model and ``fre pp``
postprocesses the history output and runs diagnostic analysis scripts. Model running is slated for
the FRE 2026.03 release; continue to use FRE Bronx frerun in the meantime.

YAML Framework
==============
In order to utilize these FRE tools, a distributed YAML structure is required. This framework includes a main model yaml, a compile yaml, a platforms yaml, and post-processing yamls. Throughout the compilation and post-processing steps, combined yamls that will be parsed for information are created. Yamls follow a dictionary-like structure with ``[key]: [value]`` fields.
Yaml files
==========

About Yaml files
----------------
FRE uses a distributed set of YAML configuration files that together define a model experiment.
Four types of files are required:

- **Model yaml** — the hub file that defines reusable properties, paths to the other yamls, and
the list of experiments
- **Compile yaml** — source code repositories, component dependencies, and build instructions
- **Platforms yaml** — site- and compiler-specific settings for each target computing environment
- **Post-processing yamls** — postprocess component definitions, settings, and switches

The compile and platforms yamls are described in detail under `Build a model`_. The
post-processing yamls are described under `Postprocessing`_.

Yaml schema
-----------
Machine-readable schemas for all FRE YAML files are maintained in the
`NOAA-GFDL/gfdl_msd_schemas <https://github.com/NOAA-GFDL/gfdl_msd_schemas/tree/main/FRE>`_
repository. To validate a yaml against the schema locally:

.. code-block:: console

fre yamltools validate-yaml -y model.yaml -e experiment_name -p platform -t target

Variables in FRE Yaml files
----------------------------
FRE expands several types of variables when processing yaml files.

*Environment variables* — Standard shell environment variables such as ``$USER``, ``$HOME``,
and ``$ARCHIVE`` are expanded by FRE at runtime.

*User-defined properties* — The ``fre_properties`` block in the model yaml defines YAML anchors
(``&name value``) that can be referenced elsewhere with ``*name``:

.. code-block:: console

fre_properties:
- &group_name "am5"
- &version "2024.01"

*The* ``!join`` *constructor* — Combines multiple anchors and strings into a single value,
useful for building directory paths:

.. code-block:: console

pp_dir: !join [/archive/$USER/, *group_name, /, *version, /, pp]

Directories
-----------
The ``directories:`` key in the post-processing settings yaml sets the paths for all managed FRE
output:

.. code-block:: console

directories:
history_dir: /path/to/raw/model/history/tarballs (required)
pp_dir: /path/to/postprocessed/output (required)
ptmp_dir: /path/to/scratch/working/space (required)
refined_history_dir: /path/to/refinediag/output (required if using refineDiag)
analysis_dir: /path/to/analysis/script/output (required if running analysis scripts)

Yaml Formatting
---------------
Expand All @@ -27,20 +89,22 @@ Post-processing Yamls
---------------------
.. include:: usage/yaml_dev/pp_yaml.rst

Build FMS model
===============
Build a model
=============
.. include:: usage/build_fms_model.rst
.. include:: usage/guides/fre_make_guide.rst

Run FMS model
=============
Check back in the latter half of 2025 or so.
Running an experiment
=====================
Running experiments via fre-cli is slated for the FRE 2026.03 release. In the meantime, continue
to use the legacy FRE Bronx ``frerun`` tool; documentation is available at the
`FRE Bronx running an experiment <https://sites.google.com/noaa.gov/oar-gfdl-msd-docs/fre-documentation/fre-documentation/running-an-experiment>`_ page.

Postprocess FMS history output
==============================
Postprocessing
==============
.. include:: usage/postprocess.rst
.. include:: usage/guides/fre_pp_guide.rst

Generate data catalogs
======================
.. include:: tools/catalog.rst
User plugins
============
.. include:: usage/user_plugins.rst
Loading
Loading