Enable River Routing (GEOSroute_GridComp) for Land in GEOS GCM for AMIP Runs

[zyj8881357]

This is a companion PR to GEOS-ESM/GEOSgcm_GridComp#1450.

The changes in this PR are straightforward configuration updates:

1. Added configuration entries for the route restart file in AGCM.rc.tmpl.

2. Added route diagnostic outputs in HISTORY.AGCM.rc.tmpl.

Expect further updates to this PR before merging.

@gmao-rreichle @biljanaorescanin @weiyuan-jiang

[zyj8881357]

The exact setting for TILING_FILE (tile.data vs. tile.bin) in AGCM.rc.tmpl remains unsettled.

[gmao-rreichle]

I don't think we want the ref_time to be 21z for daily averages. I see that 21z is used in some collections and remains unspecified in others. Not sure what the default behavior is when ref_time is unspecified. Could be 0z, or could be the start time of the simulation. In any case, when we output daily averages in a longer simulation, we probably want the daily averages aligned with full UTC days.

cc: @biljanaorescanin @zyj8881357

[biljanaorescanin]

I think the 21z setting was probably chosen to follow the usual GCM development run convention, since those runs typically start at 21z. In practice, most development runs also retain only monthly collections because of storage limitations, so the daily averaging alignment may not have received much attention.

[zyj8881357]

I tried to remove the ref_time, or set it to 0z. The simulation cannot run with conventional restart time (21z), with errors like:

ERROR: History and Averaging coupler alarms disagree.Check REF_TIME for tavg24_1d_route
pe=00000 FAIL at line=02670 MAPL_HistoryGridComp.F90 <Errors in collections REF_TIME. For details, see above.>

That means users need to edit the ref_time based on their own restart time. I think maybe the 21z is good here for being consistent with others and the convention?

[mathomp4]

Ooooooh. I might need to bring in @bena-nasa and @atrayano.

ref_time in HISTORY.rc is always confusing. My rule of thumb (that I think @atrayano taught me) is that you want ref_time to be the same hhmmss that your restarts are. A "missing" ref_time defaults to 00z.

Since most users of HISTORY.AGCM.rc.tmpl are running with restarts at 21z I think by default we've always set ref_time to 21z.

But that is an assumption. If your restarts are at 00z, then you'd want it at 00z.

I can't quite remember what the effects are if your ref_time has a mismatch with your restarts. I know your first history output from that collection will be "wonky" in that it might be missing some data it could have averaged?

Hopefully @bena-nasa and @atrayano can chime in.

[gmao-rreichle]

Thanks, @mathomp4. The question for me is how we can get daily averages that run from 0z to 0z directly out of HISTORY, independent of the restart time. Maybe that's impossible, and for 21z the solution is to write 3-hourly averages that are then post-processed into daily averages.

[mathomp4]

I chatted with @bena-nasa and he says there are some subtleties with this that I did not know about. He'll add his thoughts later today.

[biljanaorescanin]

The exact setting for TILING_FILE (tile.data vs. tile.bin) in AGCM.rc.tmpl remains unsettled.

So who decides what happens here?
For route we need this line to be tile.data and not tile.bin

GEOSgcm_App/AGCM.rc.tmpl

Line 893 in e6121fd

TILING_FILE: tile.bin

@mathomp4 both @weiyuan-jiang and @gmao-rreichle said this was planned to be changed but I didn't hear that from anyone. Am I missing something here?

[gmao-rreichle]

The exact setting for TILING_FILE (tile.data vs. tile.bin) in AGCM.rc.tmpl remains unsettled.

So who decides what happens here? For route we need this line to be tile.data and not tile.bin

GEOSgcm_App/AGCM.rc.tmpl

Line 893 in e6121fd

TILING_FILE: tile.bin

@mathomp4 both @weiyuan-jiang and @gmao-rreichle said this was planned to be changed but I didn't hear that from anyone. Am I missing something here?

Here's what I think happened, but not sure: tile.bin is now on develop. This change broke route, but route wasn't on develop when the tile.bin change was merged, so there was nothing wrong with merging the tile.bin change.

I assume route explicitly uses the text version of the tile file somewhere. If that's the case, route (GCM GC#1450) should be changed to use the bin or, probably better, the nc4 version of the tile file.

cc: @mathomp4 @biljanaorescanin @zyj8881357 @weiyuan-jiang

[gmao-rreichle]

@biljanaorescanin pointed out on Teams that the "tile.bin" change was done 4 years ago... Here's another hypothesis: route was integrated into GEOSldas, where the text tile file is still accessible from GEOSldas.x. But this doesn't work with gcm.x, which only sees the binary file. In any case, route needs a change somewhere. Unfortunately, it may be non-zero-diff for the LDAS...

[zyj8881357]

@biljanaorescanin pointed out on Teams that the "tile.bin" change was done 4 years ago... Here's another hypothesis: route was integrated into GEOSldas, where the text tile file is still accessible from GEOSldas.x. But this doesn't work with gcm.x, which only sees the binary file. In any case, route needs a change somewhere. Unfortunately, it may be non-zero-diff for the LDAS...

I found the tile.bin does not include the pfaf index information, but it is included in tile.data. That is why route cannot use tile.bin. Why tile.bin is not identical to tile.data? Could anyone help to make the tile.bin identical to tile.data?

[gmao-rreichle]

I found the tile.bin does not include the pfaf index information, but it is included in tile.data. That is why route cannot use tile.bin. Why tile.bin is not identical to tile.data?

I'm guessing whoever wrote the conversation from tile.data to tile.bin skipped whatever the GCM wasn't using at the time. The pfaf index info is needed only for routing, which didn't exist at the time.

Could anyone help to make the tile.bin identical to tile.data?

Before we mess with tile.bin, we need to establish why we cannot use the nc4 tile file for the purpose of routing.

[zyj8881357]

I found the tile.bin does not include the pfaf index information, but it is included in tile.data. That is why route cannot use tile.bin. Why tile.bin is not identical to tile.data?

I'm guessing whoever wrote the conversation from tile.data to tile.bin skipped whatever the GCM wasn't using at the time. The pfaf index info is needed only for routing, which didn't exist at the time.

Could anyone help to make the tile.bin identical to tile.data?

Before we mess with tile.bin, we need to establish why we cannot use the nc4 tile file for the purpose of routing.

Do you mean using nc4 tile file only for the Route_GridComp while keep using tile.bin for rest, or totally replacing tile.bin with nc4 tile file?

[zyj8881357]

I found the tile.bin does not include the pfaf index information, but it is included in tile.data. That is why route cannot use tile.bin. Why tile.bin is not identical to tile.data?

I'm guessing whoever wrote the conversation from tile.data to tile.bin skipped whatever the GCM wasn't using at the time. The pfaf index info is needed only for routing, which didn't exist at the time.

Could anyone help to make the tile.bin identical to tile.data?

Before we mess with tile.bin, we need to establish why we cannot use the nc4 tile file for the purpose of routing.

I just tested that the AGCM (with routing) can run with nc4 tile file.

[gmao-rreichle]

Do you mean using nc4 tile file only for the Route_GridComp while keep using tile.bin for rest, or totally replacing tile.bin with nc4 tile file?

Just the routing part would be sufficient for now. We'll have to work with the GCM and DAS groups and the SI team to make the full transition happen. (It's already done for GEOSldas.)

[mathomp4]

Do you mean using nc4 tile file only for the Route_GridComp while keep using tile.bin for rest, or totally replacing tile.bin with nc4 tile file?

Just the routing part would be sufficient for now. We'll have to work with the GCM and DAS groups and the SI team to make the full transition happen. (It's already done for GEOSldas.)

Yes. At the moment GEOSgcm is like this:

• Current GEOSgcm main: when you make an experiment, you only get the text tile.data and binarytile.x makes tile.bin at run time
• Soon to be GEOSgcm with Linkbcs Standalone Script #823 from @sshakoor1: Working with @biljanaorescanin we did a sweep of many of the BCs and pre-made tile.bin files for those. So when you run that, we link in a tile.bin so users don't need to do extra calculation (which can be expensive at high res)

From my looking at the BCs, I think only the v14 BCs have the NC4 file along with bin and text. When we move to v14, then we have nc4 tile files.

I know @sdrabenh is doing v14 tests now (as he exposed issues in #823 ). So, the question is: would this need to come in before v14 is default? Then one could "wait" I suppose. But it would mean a user could only run Route with v14 which might be handicapping science too much.

--

Also, I just made #905 as I realize linkbcs.py probably doesn't know about nc4 tile files. Since well, I didn't until now.

[gmao-rreichle]

@mathomp4 , @zyj8881357 :

Requiring v14 and newer bcs to run route may be ok for the GCM. I imagine that when we want to work within an existing system (e.g., M21C), we would use GEOSldas to run route.

At the root of the problem is that GEOSldas.x has been using the ASCII file directly, which means that the move to the nc4 tile file is not zero-diff. We "solved" this problem by making GEOSldas use the nc4 tile file by default when it is present, which is why we didn't provide it for v12 or older bcs (I think).

Having said that, I think we need to retrofit older bcs with the nc4 tile file and find a way to address the default in GEOSldas differently. We already have an rc variable TILE_FILE_FORMAT: that lets us force the use of the txt file. By editing this variable in the rc files of the LDAS nightly tests, we should be able to maintain them 0-diff for now while also retrofitting the older bcs.

This also brings up the issue that the LDAS tests should probably move to newer bcs...

cc: @biljanaorescanin @weiyuan-jiang