AWI-ESM3#1479
Open
JanStreffing wants to merge 461 commits into
Open
Conversation
…statements - Replace UKK0, UKK1, UKK13, UKK111, UKK999 debug markers with structured [TAG] messages - Add [PISM2ESM], [BUILD_SUBMESH], [MESH_FLAGS], [ICEBERG_FORCING] prefixed output - Improve mesh_flags.py print statements with progress indicators and data summaries - Add test.sh improvements: clean option, portable paths, auto-copy grid files - Add fallback for ice.griddes in COUPLE_DIR for test environments - All improvements maintain backward compatibility
- Document new components: shell scripts, Python, and Fortran - Include ASCII call tree showing execution flow - Describe data flow from PISM through processing to FESOM - List PISM mask values and their treatment - Document workflow execution steps - List HPC environment details
Implements ice-to-atmosphere coupling using ocp-tool for dynamic orography updates: - New coupling function: coupling_ice2oifs.functions * Calls ocp-tool to regrid ice mask/thickness to OIFS spectral grid * Generates ICMGGINIUA file with orography anomalies * Supports both elev_lo (ECHAM) and PISM-based reference orography - Environment setup: env_pism.py * Added OCP-tool environment variables (pool_dir, resolution, expid, etc.) * Automatically populated from ESM-Tools config - Coupling updates: coupling_ice2fesomUKK.functions * Enhanced submesh generation for ocp-tool compatibility * Creates mesh.nc with proper node/element metadata - Orography handling: coupling_pism2atmosphere.functions * Added orog_reference_pism flag (0=PISM-based, 1=elev_lo-based) * Supports different reference surfaces for anomaly calculation - Test script: test.sh * Integrated ice2oifs coupling test * Hardcoded OCP-tool variables for standalone testing - Setup config: awiesm3.yaml * Configuration updates for AWIESM3 setup Tested with TCO95 OIFS resolution and CORE2 FESOM mesh.
… vegetation coupling
12 monthly, 4 daily, 1 yearly output file for new LPJG CMIP variables
…pjg_default_cmip7_output.ins.j2
…s into feat/awiesm3-v3.4-co2
Default output_step_freq is 6 (6h cadence), which causes IFS accumulated flux fields (ssrd, slhf, sshf, cp, sf, ...) to advance in XIOS only every 6h while field_def uses freq_op=1h with a /3600 divisor. Result: radiation and precipitation means are inflated by ~6x. Override output_step_freq to 1h under the cmip7 mip block so NFRPOS/NFRHIS match the XIOS freq_op.
Add with_xios switch to fesom-2.7 (default false) and awiesm3 setup: when true, compile FESOM with -DFESOM_WITH_XIOS=ON against XIOS_ROOT and reshape xios nproc/omp to cores_per_node/8 with 8 threads. Update core2.yaml xios nproc/omp to 32/8. Add core2_2m.yaml runscript (2-month smoke test) and CMIP7 xios XML bundle under namelists/oifs.
Set domain name/dim_i_name for nodes/elements so output uses nod2/elem instead of XIOS defaults. Add time_counter_name=time and time_counter=exclusive to file_definition to keep legacy time variable name.
…_version block (fix duplicate key)
Move the PISM<->FESOM dynamic-mesh runtime coupling out of the runscript
into awiesm3.yaml, gated on a new general.interactive_mesh boolean rather
than the model version -- so any awiesm3 version can enable it without a
dedicated -is variant. Only the -is *build* target (couplings list +
meshpart_flag) stays version-gated.
configs/setups/awiesm3/awiesm3.yaml:
- general.interactive_mesh: false (feature switch, default off -> no
effect on non-coupled develop/develop-cc runs)
- fesom/oifs choose_general.interactive_mesh: true -> the couple_in /
couple_namcouple / couple_out + just_copy subjobs and
regen_oasis_weights, script_dir via ${general.esm_couplings_dir}
runscripts/.../awiesm3_dyn_ocean_core3.yaml:
- drop both workflow blocks, regen_oasis_weights and general.script_dir
(~55 lines) for a single interactive_mesh: true
Verified with 'esm_runscripts -c -t prepcompute': the finished_config
merges the coupling subjobs (incl. fix_namcouple_feom_dim, now defined
only in awiesm3.yaml) identically to the old inline runscript.
Incidental (already in the working tree): awiesm3.yaml also lands pending
with_co2_concdriven config; the runscript also carries the base_dir
setup-version subdir tweak.
…ate files
chunky_parts built the chunk_date / model-.date / finished_config paths by raw
string concatenation of general.base_dir at SimulationSetup init -- BEFORE
esm_parser resolves variables. A base_dir containing ${user} was probed
verbatim; the silent isfile() miss made iterative coupling fall back to
model1/chunk 1, so the model handoff never fired (awiesm3 self-resubmitted to
its final_date; pism never ran). _early_resolve() now expands ${...} against
general (env fallback) in all three path builders.
develop-is runscripts: shared literal awiesm3-develop-is base_dir component
(back on ${user} style now that it resolves), WEIGHTED_ICE=.false. + raw-ist
rstos (weighted-ist rollback), WAM maxocean-s85 tables, Antarctic PISM config.
The atmosphere's ice-tile skin was hard-pinned to the remapped FESOM ice temperature (vlamsk: LNEMOSICOUP default sets the sea-ice tile skin conductivity to 1e10), so the skin was never solved and marginal-ice-zone cells emitted multi-kW/m2 spurious fluxes -- the cubasen/vsurf/voskin crash family. New mode (validated, movcav33: year crash-free, kW events at real tiles 1410-1930 -> 0): - o2a exchange gains sit_feom -> A_Ice_thickness (effective grid-mean m_ice; OIFS divides by the received ice fraction on use) - runscript NAMMCC: LNEMOSICOUP=.false. (skin genuinely solved by the SEB), LNEMOLIMTEMP=.false. (no YTL overwrite; ice temperature OIFS-prognostic), LNEMOLIMTHK=.true. (slab conduction through the coupled thickness + snow) - fesom namelist.ice latm_owns_ist=.true.: growth budget takes the atmosphere flux directly (ECHAM convention); FESIM ist internal-only - chunk-1 ini: clean reference-derived ice state + rstos with sit_feom
The DIRECT (oce2ice_method) synthesis delivers PISM's given-route fields (shelfbtemp [degC], shelfbmassflux [kg m-2 s-1] = FESOM's own cavity melt), but the couplers block still invoked '-ocean th', whose three-equation init reads the 3D theta/salinity companions in the same file and rejects FESOM's decreasing depth axis. Pair the coupler with the route: couplers.ocean th -> given (-ocean_given_file), and have the DIRECT delivery select only the given-route fields so no stray 3D axis ships at all.
… windows - Remap the categorical PISM mask with remapnn: bilinear averaging of categories rounds few-cell islands to ocean and deletes them from the submesh (a decade of PISM thinning Elephant Island's cap exposed this: the drowned island drove a runaway coastal jet and a barotropic blowup at the Peninsula tip). remapnn extrapolates, so the mask=5 outside-domain marker is rebuilt from a bilinear domain indicator. - pad_dry_tracer_levels: fill dry tracer levels of the remapped restart with the nearest wet value so anything touching them sees inert values. - Extend the iterative-coupling windows to 10 awiesm3 chunks (1900-1909) and 100 PISM years.
PISM's Mask.hh: MASK_ICE_FREE_BEDROCK=0. The classifier defaulted unassigned categories to ocean and only mapped mask==1 to land (a convention that never occurs), so any island whose ice cap melts away completely would be deleted from the submesh even with the nearest-neighbour mask remap -- the first such cell (on Elephant Island, bed +4 m, thk 0) exists in the current geometry. Bare bedrock above sea level is land regardless of ice.
… by <=2 elements A reorganizing ice front (simultaneous retreat and grounding advance in one PISM decade) leaves fragile coastal slivers in the carved submesh: nodes held by only 1-2 elements, including deep-cavity columns on a knife edge next to open water. These drove a barotropic blowup at the western Ross front 16 days into the leg after the second increment. The sweep folds into the existing element-removal convergence loop and iterates until the coastline is clean. Validated: increments 2-12 (120 PISM years) subsequently ran without a single FESOM blowup. Also drop the temporary crash-forensics XIOS daily file group.
…des held by <=2 elements" This reverts commit 680daa5.
…tion guard mesh_flags: clamp the cavity roof to within RATE_LIMIT_DRAFT (default 75 m) of the previous submesh's roof. A reorganizing PISM front can move the roof by O(1 km) in one decade (the 1902 increment carried 1196 m swings, opening 13 levels at a node in one step next to a 506-node grounding advance); FESOM absorbs such steps badly. Offline harness on the saved 1902 flags: 1357 nodes clamped, max change 1196->75 m, and the resulting mesh RETAINS the cavities (2221 nodes vs 1864 unclamped). Fragility itself is not the failure mode: meshes that ran fine carry the same ~3450-node <=2-element census as the crashed one. coupling functions: abort the leg if the reduced submesh keeps <50% of the classifier-flagged cavity nodes - the guard that would have caught the hygiene-sweep amputation immediately.
It makes the ocean's cavity geometry deliberately lag the ice model's actual geometry - a coupler-imposed inconsistency between components. The large increments must be addressed at their source (the melt bias) or by numerical treatment of the post-increment transient, not by distorting the coupled geometry. The cavity-amputation abort guard in the coupling functions stays.
Contributor
|
Hi @JanStreffing, is it okay if I merge |
Contributor
Author
|
Yes, no problem |
… coupling) Applies the CUDA-disable fix from 4c64269 to the pism/debm coupling too; that file still carried the broken ;-joined sed. atlas 0.39 vendors the `hic` subproject, which declares the global ENABLE_CUDA feature with no CONDITION guard. eckit 1.18.8 declares the same global feature without a DEFAULT, seeding ENABLE_CUDA=ON into the CMake cache before disabling it for itself via CONDITION HAVE_EXPERIMENTAL. hic then inherits the cached ON and hard-fails on find_package(CUDAToolkit REQUIRED) -> "Could not find nvcc". ATLAS_ENABLE_CUDA=OFF does not help: that is the atlas-scoped variable, distinct from the global one hic reads. Seed the unprefixed ENABLE_CUDA=OFF via coupling_changes so ecbundle_set writes it to the cache first. Split into two separate sed commands rather than one ;-joined script: esm_master runs coupling_changes through command.split(";") before shlex.split(), so a ; inside the quoted sed breaks quote balancing. The leading delete keeps it idempotent under recomp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mandresm
requested changes
Jul 21, 2026
mandresm
left a comment
mandresm
left a comment
Contributor
There was a problem hiding this comment.
Hi @JanStreffing, I am going to be doing this PR review in chunks. Please, see my suggestions for the FESOM files.
| config: config | ||
| cvmix: cvmix | ||
| dyn: dyn | ||
| forcing: forcing |
Contributor
There was a problem hiding this comment.
Suggested change
| forcing: forcing | |
| nml_forcing: nml_forcing |
This prevents an issue with the forcing key under some edge cases (same as in fesom-2.6.yaml.
| config: "${namelist_dir}/namelist.config" | ||
| cvmix: "${namelist_dir}/namelist.cvmix" | ||
| dyn: "${namelist_dir}/namelist.dyn" | ||
| forcing: "${namelist_dir}/namelist.forcing" |
Contributor
There was a problem hiding this comment.
Suggested change
| forcing: "${namelist_dir}/namelist.forcing" | |
| forcing_nml: "${namelist_dir}/namelist.forcing" |
Same a s above. This prevents an issue with the forcing key under some edge cases (same as in fesom-2.6.yaml.
| config: "namelist.config" | ||
| cvmix: "namelist.cvmix" | ||
| dyn: "namelist.dyn" | ||
| forcing: "namelist.forcing" |
Contributor
There was a problem hiding this comment.
Suggested change
| forcing: "namelist.forcing" | |
| forcing_nml: "namelist.forcing" |
Same a s above. This prevents an issue with the forcing key under some edge cases (same as in fesom-2.6.yaml.
|
|
||
| further_reading: | ||
| - fesom/fesom.forcing.yaml | ||
| - fesom/fesom.env.yaml |
Contributor
There was a problem hiding this comment.
Suggested change
| - fesom/fesom.env.yaml | |
| - fesom/fesom.env.yaml | |
| - fesom/fesom.tripyview.yaml |
Tipyview was missing here, is it intended?
Comment on lines
+274
to
+277
| outdata_sources: | ||
| fesom: "*.fesom.*.nc" | ||
| outdata_targets: | ||
| fesom: "*.fesom.*.nc" |
Contributor
There was a problem hiding this comment.
Suggested change
| outdata_sources: | |
| fesom: "*.fesom.*.nc" | |
| outdata_targets: | |
| fesom: "*.fesom.*.nc" | |
| outdata_sources: | |
| fesom: "*.fesom.*.nc" | |
| fesom_xios: "*.fesom_${start_date!syear}-${end_date!syear}.nc" | |
| fesom_reg_xios: "*.fesom.reg_${start_date!syear}-${end_date!syear}.nc" | |
| outdata_targets: | |
| fesom: "*.fesom.*.nc" | |
| fesom_xios: "*.fesom.${start_date!syear}.nc" | |
| fesom_reg_xios: "*.fesom.gr.${start_date!syear}.nc" |
See my comment below
Comment on lines
+568
to
+611
| choose_with_xios: | ||
| true: | ||
| # XIOS writes "<var>.fesom_<ystart>-<yend>.nc" into the work | ||
| # dir. Collect those alongside the legacy pattern so tidy | ||
| # copies them into outdata/fesom, then a post-tidy subjob | ||
| # renames them to the old "<var>.fesom.<year>.nc" scheme. | ||
| add_outdata_sources: | ||
| fesom_xios: "*.fesom_*.nc" | ||
| add_outdata_targets: | ||
| fesom_xios: "*.fesom_*.nc" | ||
|
|
||
| scripts_files: | ||
| rename_xios: rename_xios | ||
| rename_xios_reg: rename_xios_reg | ||
| scripts_sources: | ||
| rename_xios: "${esm_configs_dir}/components/fesom/rename_xios_fesom.sh" | ||
| rename_xios_reg: "${esm_configs_dir}/components/fesom/rename_xios_fesom_reg.sh" | ||
| scripts_targets: | ||
| rename_xios: "${general.experiment_scripts_dir}/${model}/rename_xios_fesom.sh" | ||
| rename_xios_reg: "${general.experiment_scripts_dir}/${model}/rename_xios_fesom_reg.sh" | ||
|
|
||
| workflow: | ||
| next_run_triggered_by: tidy | ||
| subjobs: | ||
| Rename_XIOS_FESOM: | ||
| batch_or_shell: shell | ||
| order_in_cluster: concurrent | ||
| run_after: tidy | ||
| script_dir: ${general.experiment_scripts_dir}/${model} | ||
| script: "rename_xios_fesom.sh ${fesom.experiment_outdata_dir}" | ||
| nproc: 1 | ||
| # Sibling renamer for regridded XIOS output: | ||
| # <var>.fesom.reg_<ystart>-<yend>.nc -> <var>.fesom.gr.<ystart>.nc | ||
| # Disjoint glob from Rename_XIOS_FESOM (this one needs | ||
| # ".fesom.reg_" literally), safe to run concurrently. | ||
| # Idempotent: skips files already in target form, no-ops | ||
| # if no regridded files are present (eg fesom.regrid_surface_output=false). | ||
| Rename_XIOS_FESOM_REG: | ||
| batch_or_shell: shell | ||
| order_in_cluster: concurrent | ||
| run_after: tidy | ||
| script_dir: ${general.experiment_scripts_dir}/${model} | ||
| script: "rename_xios_fesom_reg.sh ${fesom.experiment_outdata_dir}" | ||
| nproc: 1 |
Contributor
There was a problem hiding this comment.
Suggested change
| choose_with_xios: | |
| true: | |
| # XIOS writes "<var>.fesom_<ystart>-<yend>.nc" into the work | |
| # dir. Collect those alongside the legacy pattern so tidy | |
| # copies them into outdata/fesom, then a post-tidy subjob | |
| # renames them to the old "<var>.fesom.<year>.nc" scheme. | |
| add_outdata_sources: | |
| fesom_xios: "*.fesom_*.nc" | |
| add_outdata_targets: | |
| fesom_xios: "*.fesom_*.nc" | |
| scripts_files: | |
| rename_xios: rename_xios | |
| rename_xios_reg: rename_xios_reg | |
| scripts_sources: | |
| rename_xios: "${esm_configs_dir}/components/fesom/rename_xios_fesom.sh" | |
| rename_xios_reg: "${esm_configs_dir}/components/fesom/rename_xios_fesom_reg.sh" | |
| scripts_targets: | |
| rename_xios: "${general.experiment_scripts_dir}/${model}/rename_xios_fesom.sh" | |
| rename_xios_reg: "${general.experiment_scripts_dir}/${model}/rename_xios_fesom_reg.sh" | |
| workflow: | |
| next_run_triggered_by: tidy | |
| subjobs: | |
| Rename_XIOS_FESOM: | |
| batch_or_shell: shell | |
| order_in_cluster: concurrent | |
| run_after: tidy | |
| script_dir: ${general.experiment_scripts_dir}/${model} | |
| script: "rename_xios_fesom.sh ${fesom.experiment_outdata_dir}" | |
| nproc: 1 | |
| # Sibling renamer for regridded XIOS output: | |
| # <var>.fesom.reg_<ystart>-<yend>.nc -> <var>.fesom.gr.<ystart>.nc | |
| # Disjoint glob from Rename_XIOS_FESOM (this one needs | |
| # ".fesom.reg_" literally), safe to run concurrently. | |
| # Idempotent: skips files already in target form, no-ops | |
| # if no regridded files are present (eg fesom.regrid_surface_output=false). | |
| Rename_XIOS_FESOM_REG: | |
| batch_or_shell: shell | |
| order_in_cluster: concurrent | |
| run_after: tidy | |
| script_dir: ${general.experiment_scripts_dir}/${model} | |
| script: "rename_xios_fesom_reg.sh ${fesom.experiment_outdata_dir}" | |
| nproc: 1 | |
| choose_with_xios: | |
| true: | |
| # dir. fesom_xios and fesom_reg_xios file labels account for | |
| # this different name pattern and ensures that the files are | |
| # renamed when copied/moved to the outdata dir with the legacy | |
| # pattern. | |
| add_outdata_files: | |
| fesom_xios: fesom_xios | |
| fesom_reg_xios: fesom_reg_xios |
I don't think additional scripts are needed for this task, unless you are modifying the files, which I don't think you are doing it. With the suggested changes here and a the previous suggestion you should get identical file naming in the <expid>/outdata/fesom directory with bit identical files (I have tested both approaches, the shell scripts ones and the one in the suggestions and they do give identical files).
Contributor
There was a problem hiding this comment.
I think you can remove this script
Contributor
There was a problem hiding this comment.
I think this script can be removed
The ice-front vertical-coordinate seam -- zstar stretching the open-ocean layers by the free surface while the rigid-lid cavity keeps them nominal -- drove the barotropic cavity spin-up that was re-injected at every mesh increment. linfs (fixed layers everywhere) removes it by construction: the raw remap under linfs decays to ~1.7 cm/s with no hnode surgery, and the coupled movcav37 run holds 1-2 cm/s across all 9 increments. Set which_ale=linfs in both moving-cavity chunk blocks (chunk 1 and chunk>=2).
…ng cavity When the moving cavity advances over open water, the remap carried each newly-iced node's own open-ocean column under the new ice base -- up to +6.6 C of thermal driving at the roof -- and the old freezing cap then over-cooled the deep part to the pressure freezing point (-5.35 C at 4.5 km, below FESOM's -5 C reject: the 1904 blowup). Instead seed each open->cavity column from the nearest column that is cavity in both meshes (cold, cavity-appropriate ice-shelf water, depth-matched); no cap needed. Ice-base warm columns 61->9, peak driving +6.6->+0.8 C. Keep a freezing-point floor as a last-resort backstop. Also drop the dead experimental machinery from the investigation: the env-gated fills smooth_cavity_fill and ambient_reservoir_fill (both proven ineffective), and the dormant geostrophic-overwrite branch of geostrophic_init_changed (renamed init_velocity_changed, keeping the active AB-reset + seam-smoothing). Output verified byte-identical on a real increment before/after the cleanup.
Point spinup_pismPI.yaml and spinup_pismAnt.yaml at the renamed input pool (/work/ab0246/a270092/input/pism/).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With the CMIP7 PI control starting, maybe its time to think about merging?
Since we start without automatic cmorization for now, the esm_tools development is not quite over.