Development interactiv mesh awiesm3#1446
Open
JanStreffing wants to merge 1031 commits into
Open
Conversation
Default output_step_freq is 6 (6h cadence), which causes IFS accumulated flux fields (ssrd, slhf, sshf, cp, sf, ...) to advance in XIOS only every 6h while field_def uses freq_op=1h with a /3600 divisor. Result: radiation and precipitation means are inflated by ~6x. Override output_step_freq to 1h under the cmip7 mip block so NFRPOS/NFRHIS match the XIOS freq_op.
Add with_xios switch to fesom-2.7 (default false) and awiesm3 setup: when true, compile FESOM with -DFESOM_WITH_XIOS=ON against XIOS_ROOT and reshape xios nproc/omp to cores_per_node/8 with 8 threads. Update core2.yaml xios nproc/omp to 32/8. Add core2_2m.yaml runscript (2-month smoke test) and CMIP7 xios XML bundle under namelists/oifs.
Set domain name/dim_i_name for nodes/elements so output uses nod2/elem instead of XIOS defaults. Add time_counter_name=time and time_counter=exclusive to file_definition to keep legacy time variable name.
…_version block (fix duplicate key)
…ace_conditions fix issues in #1457
pycmor now recomputes plev hur from ta+hus with CMIP7 phase-dependent saturation vapour pressure (water >=0C, ice <0C). IFS FullPos 'r' uses mixed-phase QSAT interpolation that is not CMIP-compliant, so it is dropped from the archive and replaced by ta+hus as plev outputs. Also carries earlier ahead-of-branch additions: rsds on one plev group and 'ci' sea-ice area fraction.
All grid_ref in file_def_oifs_cmip7_spinup.xml.j2 flipped from 'regular_*' to 'reduced_*'. Added matching reduced_pl3/pl6/pl7h grid defs (same axes on reduced_gaussian domain) since only the full plev/ml/sfc had reduced variants before.
Consolidates the atmos file_group from 38 to 19 file blocks, grouped strictly by vertical level (sfc/ml/pl/pl3/pl6/pl7h), output frequency (1h/3h/6h/1d/1mo), and operation (avg/inst/max/mixed). No mixing of instant/average within a single field_group. All fields preserved. The _day_sfc_mixed block keeps per-field max/min definitions (tasmax, tasmin, sfcWindmax). The atm_remapped file_group is left untouched. Note: XIOS timeseries filenames include name_suffix, so output files change from e.g. atmos_1h_sfc_rsds_1h_sfc_<year>.nc to atmos_1h_sfc_avg_rsds_1h_sfc_avg_<year>.nc. Downstream pycmor rule regexes will need updating to the new suffix set.
Pairs with oifs branch local_combined_fixes (f605d7c). Without it OIFS aborts in SUXIOS_NAMFPC_L on the hardcoded "plev39" default.
… of regular lat-lon (gr) - grid_def.xml: add reduced_pl3, reduced_pl6, reduced_pl7h counterparts - file_def_oifs_cmip7_spinup.xml.j2: switch all 48 grid_ref="regular_*" to "reduced_*" so CMOR output matches the CMIP7 source grid specification (gn) and avoids unnecessary interpolation cost.
…s inverse. False = ON ...)
…path Align the native FESOM output list with what the CMIP7 XIOS file_def (xios_xml_cmip7/file_def_fesom.xml.j2) emits, so FESOM produces the same variable/frequency set when routed via namelist.io instead of XIOS. - Enable ldiag_trflx so utemp/vtemp/usalt/vsalt (on tracer-flux diagnostics at 3D elements) are populated; XIOS file_def requests these. - Document default dmoc_call_freq explicitly in the namelist. - Reorder io_list to mirror XIOS structure (3h / 1d / 1mo blocks, sub-grouped by 2D nodes / 2D elements / 3D nodes / 3D elements). - Move tx_sur, ty_sur from monthly to 3-hourly (matches XIOS). - Move h_ice, h_snow from monthly to daily (matches XIOS). - Rename thdgrsn -> thdgrsnw (typo; matches FESOM source CASE name in io_meandata.F90). - Add prec, snow, runoff (in XIOS, missing from namelist). - Add 'dMOC' meta-entry: without it, dMOC streams fall through to FESOM's default annual path (io_meandata.F90); XIOS emits them monthly, so this aligns frequencies. - Drop 'otracers' (not in XIOS file_def). Auto-emitted via flags, not listed here: pbo, opottemptend, volo, soga, thetaoga, siarean/siareas/ siextentn/siextents/sivoln/sivols via ldiag_cmor=.true. utemp, vtemp, usalt, vsalt via ldiag_trflx=.true.
… are correct The retired static field_def_lpjg_safe.xml (removed in cea79d4) hard-coded the de-accumulation period to 21600 s (6h freq_op cadence) instead of NFRHIS*TSTEP, making every OpenIFS accumulated flux fed to LPJG 6x too low -> offline LPJG spin-up ran at ~1/6 NPP (boreal ~0). Commit the replacement jinja template field_def_lpjg_safe.xml.j2 (driven by oifs.flux_accumulation_period) and default that variable to ${oifs.time_step} in the oifsamip setup, so no oifsamip+LPJG runscript can reintroduce the bug (override only if NFRHIS!=1).
Discover the dynamic-mesh coupling paths instead of hardcoding them in
every runscript.
env_fesom.py:
- locate ocp-tool via importlib (it is a required_plugin esm_tools
pip-installs) instead of a hardcoded software path
- resolve the driver/worker conda envs by name via 'conda env list'
(driver name read from ocp-tool's environment.yaml; worker 'ece4',
overridable via ocp_weightgen_worker_env)
- discover pyfesom2 (an ocp-tool dependency) via importlib; drop the
stale PYFESOM_PATH hardcode (pointed at a foreign home that no longer
exists)
- derive oasis_build_path / oasis_rmp_template_dir / ocp_template_config
from model_dir / pool_dir / resolution
- keep ocp_weightgen_threads default at 64
All values still honour an explicit per-runscript override.
runscripts/awiesm3/develop-is:
- script_dir -> ${general.esm_couplings_dir} (relative to the install)
- base_dir unified to /work/ab0246/${user}/runtime/ across the three
scripts (+ user: !ENV ${USER} where it was missing)
- drop the now-discovered ocp-tool/OASIS path block from the fesom section
Move the PISM<->FESOM dynamic-mesh runtime coupling out of the runscript
into awiesm3.yaml, gated on a new general.interactive_mesh boolean rather
than the model version -- so any awiesm3 version can enable it without a
dedicated -is variant. Only the -is *build* target (couplings list +
meshpart_flag) stays version-gated.
configs/setups/awiesm3/awiesm3.yaml:
- general.interactive_mesh: false (feature switch, default off -> no
effect on non-coupled develop/develop-cc runs)
- fesom/oifs choose_general.interactive_mesh: true -> the couple_in /
couple_namcouple / couple_out + just_copy subjobs and
regen_oasis_weights, script_dir via ${general.esm_couplings_dir}
runscripts/.../awiesm3_dyn_ocean_core3.yaml:
- drop both workflow blocks, regen_oasis_weights and general.script_dir
(~55 lines) for a single interactive_mesh: true
Verified with 'esm_runscripts -c -t prepcompute': the finished_config
merges the coupling subjobs (incl. fix_namcouple_feom_dim, now defined
only in awiesm3.yaml) identically to the old inline runscript.
Incidental (already in the working tree): awiesm3.yaml also lands pending
with_co2_concdriven config; the runscript also carries the base_dir
setup-version subdir tweak.
…ate files
chunky_parts built the chunk_date / model-.date / finished_config paths by raw
string concatenation of general.base_dir at SimulationSetup init -- BEFORE
esm_parser resolves variables. A base_dir containing ${user} was probed
verbatim; the silent isfile() miss made iterative coupling fall back to
model1/chunk 1, so the model handoff never fired (awiesm3 self-resubmitted to
its final_date; pism never ran). _early_resolve() now expands ${...} against
general (env fallback) in all three path builders.
develop-is runscripts: shared literal awiesm3-develop-is base_dir component
(back on ${user} style now that it resolves), WEIGHTED_ICE=.false. + raw-ist
rstos (weighted-ist rollback), WAM maxocean-s85 tables, Antarctic PISM config.
The atmosphere's ice-tile skin was hard-pinned to the remapped FESOM ice temperature (vlamsk: LNEMOSICOUP default sets the sea-ice tile skin conductivity to 1e10), so the skin was never solved and marginal-ice-zone cells emitted multi-kW/m2 spurious fluxes -- the cubasen/vsurf/voskin crash family. New mode (validated, movcav33: year crash-free, kW events at real tiles 1410-1930 -> 0): - o2a exchange gains sit_feom -> A_Ice_thickness (effective grid-mean m_ice; OIFS divides by the received ice fraction on use) - runscript NAMMCC: LNEMOSICOUP=.false. (skin genuinely solved by the SEB), LNEMOLIMTEMP=.false. (no YTL overwrite; ice temperature OIFS-prognostic), LNEMOLIMTHK=.true. (slab conduction through the coupled thickness + snow) - fesom namelist.ice latm_owns_ist=.true.: growth budget takes the atmosphere flux directly (ECHAM convention); FESIM ist internal-only - chunk-1 ini: clean reference-derived ice state + rstos with sit_feom
The DIRECT (oce2ice_method) synthesis delivers PISM's given-route fields (shelfbtemp [degC], shelfbmassflux [kg m-2 s-1] = FESOM's own cavity melt), but the couplers block still invoked '-ocean th', whose three-equation init reads the 3D theta/salinity companions in the same file and rejects FESOM's decreasing depth axis. Pair the coupler with the route: couplers.ocean th -> given (-ocean_given_file), and have the DIRECT delivery select only the given-route fields so no stray 3D axis ships at all.
… windows - Remap the categorical PISM mask with remapnn: bilinear averaging of categories rounds few-cell islands to ocean and deletes them from the submesh (a decade of PISM thinning Elephant Island's cap exposed this: the drowned island drove a runaway coastal jet and a barotropic blowup at the Peninsula tip). remapnn extrapolates, so the mask=5 outside-domain marker is rebuilt from a bilinear domain indicator. - pad_dry_tracer_levels: fill dry tracer levels of the remapped restart with the nearest wet value so anything touching them sees inert values. - Extend the iterative-coupling windows to 10 awiesm3 chunks (1900-1909) and 100 PISM years.
PISM's Mask.hh: MASK_ICE_FREE_BEDROCK=0. The classifier defaulted unassigned categories to ocean and only mapped mask==1 to land (a convention that never occurs), so any island whose ice cap melts away completely would be deleted from the submesh even with the nearest-neighbour mask remap -- the first such cell (on Elephant Island, bed +4 m, thk 0) exists in the current geometry. Bare bedrock above sea level is land regardless of ice.
… by <=2 elements A reorganizing ice front (simultaneous retreat and grounding advance in one PISM decade) leaves fragile coastal slivers in the carved submesh: nodes held by only 1-2 elements, including deep-cavity columns on a knife edge next to open water. These drove a barotropic blowup at the western Ross front 16 days into the leg after the second increment. The sweep folds into the existing element-removal convergence loop and iterates until the coastline is clean. Validated: increments 2-12 (120 PISM years) subsequently ran without a single FESOM blowup. Also drop the temporary crash-forensics XIOS daily file group.
…des held by <=2 elements" This reverts commit 680daa5.
…tion guard mesh_flags: clamp the cavity roof to within RATE_LIMIT_DRAFT (default 75 m) of the previous submesh's roof. A reorganizing PISM front can move the roof by O(1 km) in one decade (the 1902 increment carried 1196 m swings, opening 13 levels at a node in one step next to a 506-node grounding advance); FESOM absorbs such steps badly. Offline harness on the saved 1902 flags: 1357 nodes clamped, max change 1196->75 m, and the resulting mesh RETAINS the cavities (2221 nodes vs 1864 unclamped). Fragility itself is not the failure mode: meshes that ran fine carry the same ~3450-node <=2-element census as the crashed one. coupling functions: abort the leg if the reduced submesh keeps <50% of the classifier-flagged cavity nodes - the guard that would have caught the hygiene-sweep amputation immediately.
It makes the ocean's cavity geometry deliberately lag the ice model's actual geometry - a coupler-imposed inconsistency between components. The large increments must be addressed at their source (the melt bias) or by numerical treatment of the post-increment transient, not by distorting the coupled geometry. The cavity-amputation abort guard in the coupling functions stays.
The ice-front vertical-coordinate seam -- zstar stretching the open-ocean layers by the free surface while the rigid-lid cavity keeps them nominal -- drove the barotropic cavity spin-up that was re-injected at every mesh increment. linfs (fixed layers everywhere) removes it by construction: the raw remap under linfs decays to ~1.7 cm/s with no hnode surgery, and the coupled movcav37 run holds 1-2 cm/s across all 9 increments. Set which_ale=linfs in both moving-cavity chunk blocks (chunk 1 and chunk>=2).
…ng cavity When the moving cavity advances over open water, the remap carried each newly-iced node's own open-ocean column under the new ice base -- up to +6.6 C of thermal driving at the roof -- and the old freezing cap then over-cooled the deep part to the pressure freezing point (-5.35 C at 4.5 km, below FESOM's -5 C reject: the 1904 blowup). Instead seed each open->cavity column from the nearest column that is cavity in both meshes (cold, cavity-appropriate ice-shelf water, depth-matched); no cap needed. Ice-base warm columns 61->9, peak driving +6.6->+0.8 C. Keep a freezing-point floor as a last-resort backstop. Also drop the dead experimental machinery from the investigation: the env-gated fills smooth_cavity_fill and ambient_reservoir_fill (both proven ineffective), and the dormant geostrophic-overwrite branch of geostrophic_init_changed (renamed init_velocity_changed, keeping the active AB-reset + seam-smoothing). Output verified byte-identical on a real increment before/after the cleanup.
Point spinup_pismPI.yaml and spinup_pismAnt.yaml at the renamed input pool (/work/ab0246/a270092/input/pism/).
…g files The couplings/ tree (e.g. couplings/fesom/env_fesom.py and the coupling functions) was omitted from MANIFEST.in, so include_package_data left it out of sdist/wheel installs. The non-recursive package_data glob (../couplings/*) only caught top-level entries, so nested files were missing from venv installs (FileNotFoundError on couplings/fesom/env_fesom.py). Graft it like configs/ namelists/runscripts.
…e toolchain Under use_venv the esm_tools venv installs the ocp-tool toolchain (ocp_tool + pyfesom2 + eccodes) into its own Python, so OCP_TOOL_DIR (found by find_spec in that interpreter) points at the venv site-packages. But the weight-regen driver was still resolved to ocp-tool's conda env (ocp-tool2), a different Python version -- putting the venv's site-packages on that interpreter's PYTHONPATH caused a numpy ABI crash (No module named 'numpy._core._multiarray_umath'). Resolve the driver to sys.executable when running inside a venv that already carries the toolchain, so discovery and driver share one interpreter. Non-venv runs are unchanged (guarded on sys.prefix != sys.base_prefix) and keep using the conda env. No hardcoded paths; the required packages populate the venv.
…t has the toolchain" This reverts commit 9b1ae66.
…conda-native) ocp-tool's OASIS/ICMGG regen toolchain needs python-eccodes/python-cdo/basemap (conda-forge), so it can't run from the pip venv, and its configs/ + ICMGG input data aren't in the pip package. Under use_venv, find_spec landed OCP_TOOL_DIR on the useless venv copy. Set it per-machine (levante) to the real checkout; the venv still isolates esm_tools and the ocp-tool2 conda env drives the regen. '*' keeps package auto-discovery for editable/non-venv installs.
…iesm3
LCORBMD gates OpenIFS's use of the AWI shr_orb code. When off, OpenIFS falls
back to its IFS-native, absolute-date-driven orbit with present-day geometry,
which for a piControl gives both a wrong (present-day) orbit and a leap-year /
tropical-year sampling staircase in TOA insolation. The esm_tools defaults
left this OFF (43r3) or set a present-day variable_year orbit (48r1 template),
so no awicm3/awiesm3 version shipped a correct constant-1850 orbit.
Default the orbit to LCORBMD=.true., ORBMODE=fixed_year, ORBIY=1850
(piControl-correct, stationary) for:
- awicm3 v3.3.0/v3.3.1 (43r3): via choose_general.version, feeding the
existing 43r3 NAMORB injection (oifs.version is plain "43r3" for these,
so scope on the setup version instead).
- awicm3 v3.4+/develop and all awiesm3 (48r1): add a NAMORB injection to
choose_major_version:48r1 (which had none) driven by orb_switch/orb_mode/
orb_iyear, and blank the hard-coded variable_year/1990 NAMORB in the
48r1/awicm3/v3.4 fort.4 template so the injection is the single source.
Older 43r3 versions (v3.0-v3.2.x, frontiers) keep the historical default for
reproducibility. Transient/paleo runscripts still override NAMORB explicitly.
Verified by dry-resolving (esm_runscripts -c --open-run) awicm3 v3.3.0 and
v3.4.0 runscripts: both resolve NAMORB to LCORBMD=.true./fixed_year/1850.
…ardwired in FESOM) The FESOM build hardwires coupled-slab mode (ice_thermo_cpl.F90 Qatmice=-a2ihf); the latm_owns_ist ice_therm knob was removed there, so injecting it into namelist.ice would abort the Fortran namelist read. The oifs-side NAMMCC LNEMOLIMTEMP/LNEMOLIMTHK settings that select coupled-slab remain.
…ranches
All develop/develop-cc/develop-is now build the coupled-slab coupling:
- OIFS: new component version 48r1v5 -> movcav-landice+co2-concdriven (the
combined jan.streffing fork branch = coupled-slab receive + land-ice +
co2-concdriven); the three develop* couplings point at oifs-48r1v5.
- FESOM: component 2.7-main -> awiesm3-implicit-ice-surftemp (coupled-slab
ice-surftemp + sit_feom send). develop-cc keeps its recom branch override,
now carrying the coupled-slab merge.
- awiesm3.yaml: drop a stray oifs branch override ('lpj_guess_implementation+co2',
a since-merged lpj_guess branch) that mis-cloned the develop-cc OIFS repo.
Interim stopgap until FESOM PR #957 and the OIFS movcav-landice work merge to
their respective mains; tags (v3.4.2 and below) are unaffected (separate pins).
Validated: fresh get+comp in model_codes for all three, develop full 1-month
coupled run clean, develop-cc coupling handshake clean, develop-is iterative
2-chunk run progressing.
…oastline Closes the recurring OIFS voskin coastal overflow (report work-item iii, movcav34 chunk-13; campaign cmp_is_2y chunk-2). When an increment's regenerated ICMGG flips atm cells land->ocean, the A096-side fields of rstos.nc are carried over untouched, so those cells keep A_SST=0K; OIFS ingests that at its first step and the SP squaring in voskin_mod.F90:282 overflows (forrtl 72) -- deterministically, on the first physics step of the post-increment leg. sanitize_oasis_restart_atm.py reads the NEW lsm from the regenerated ICMGG and nearest-ocean-fills A_SST (and floors A_Ice_temp) at every ocean-per-new-lsm cell holding a stale land value; remap_oasis_restart now runs it after the feom-side remap. Verified: 21 cells filled, chunk-2 runs past the former crash (mstep 1920+, no voskin).
develop: TCO95-CORE3 1-month cold start (passed clean). develop-cc: same + the two settings a cold-started recom run needs: namelist_dir -> the FESOM config dir (so bin_<recom_setting>/namelist.recom resolves instead of an 11-line stub -> EOF crash), and recom_restart=.false. (else REcoM reads a nonexistent restart and the Alk/DIC garbage blows up the carbonate chemistry). dt=600 (1800 is only stable from a warm restart). develop-is: 2-chunk iterative pair (2 chunks so the inter-chunk mesh increment is actually exercised).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey @ukrebska-hub, here I have:
a) made the print statements of the test.sh a bit more pretty
b) took out some hard coded things that prevented me from running the test.sh
c) merged the most recent awiesm3 development branch
d) included step 4, the modification of OpenIFS lsm according to PISM modified FESOM mesh via ocp-tools. I decided not to use the spherelab generated masks_cst.nc. It was generated with spherelab and is missing the cavity netcdf description that I added to the pyfesom2 version of that code a while ago, which ocp-tools relies upon. I did however make the pyfesom2 version of the ascii2netcdf algorithm 10x faster. So now it is about 4x faster than spherelab. We should probably switch the singularity container over to using pyfesom2 then: FESOM/pyfesom2#234 Also paging @pgierz
Please merge into your branch when you can run on here. I'll be around Monday morning to help if needed.