How does netcdf-c recognize netCDF4 format

[kmuehlbauer]

This may be a stupid question, but the documentation on the format didn't bring me further.

In light of What is netCDF I'm interested in what makes an HDF5-file a netCDF4 file?

Obviously every file which is created via netcdf-c (using netCDF4 format) is a netCDF4 file. But how does this apply to other ways of creating, like those packages mentioned in the above link (eg. h5netcdf).

Consider adding/changing a valid netCDF4-file (created with netcdf-c) by other packages side-stepping netcdf-c with unsupported HDF5 features (eg. multiple links to a data object). What changes are needed to the original netCDF4-file to make it unmistakably clear, that the altered file is not a netCDF4 compliant file any more, even if it might still be readable via netcdf-c?

Will removal of the _NCProperties attribute be enough to make that difference?

[edwardhartnett]

Reading HDF5 files, in all their glory, is a core feature of netCDF-4. So you should not attempt to mark a HDF5 file as unreadable to netCDF-4. Instead, we should change netCDF-4 to read the file. And that has happened many times.

So not all HDF5 files can be read by netCDF at this time, but the feature needed to read the file might be added. Therefore it's not safe to say that a HDF5 file cannot be read by netCDF.

[kmuehlbauer]

Yes, I completely understand and I'm closely following the evolution of netcdf-c in that sense. Unidata is doing great work not only in that regard.

But I still do not know how to handle that particular problem. Should h5netcdf append currently invalid netcdf features to valid NetCDF-4 files without leaving any hint within the file (eg remove of _NCProperties? I'm concerned to throw the burden of handling user requests regarding those files to the unidata people.

[edwardhartnett]

Can we instead add the feature to netcdf-c? Would that be difficult?

[kmuehlbauer]

Quoting from netcdf documentation

Some HDF5 features not supported in the netCDF enhanced model and netCDF-4 format include non-hierarchical group structures, HDF5 reference types, multiple links to a data object, user-defined atomic data types, stored property lists, more permissive rules for data object names, the HDF5 date/time type, and attributes associated with user-defined types.

That's what the docs say is not supported.

For h5netcdf this boils down to variables with dtype bool as well as some enum, reference and non-string vlen types.

There are these open issue regarding enum and complex:

• NetCDF unable to read some HDF5 enums #267
• https://github.com/cf-convention/cf-conventions/issues/204

Maybe it would be a good idea to gather those features in one place. This will surely take it's time to implement and iron out every detail. Nevertheless a solution for the interim would be useful.

[DennisHeimbigner]

Removing _NCProperties is not enough since there are old netcdf-4 files out there that predate
that attribute. There is an attribute that you can read by name `("_IsNetcdf4", I think) that uses
some heuristics to decide if a file looks like it was created by libnetcdf (independent of _NCProperties).
The function that computes the heuristic is called "NC_isnetcdf4" and is in the file netcdf-c/libhdf5/nc4hdf.c.

The question is somewhat ill-posed. In some sense, if the file is readable by libnetcdf, then it
could be considered a netcdf file.
I suspect the issue is really "was this file written by libnetcdf?". Aside from the heuristic, the answer
is encoded in _NCProperties. If that attribute not not show netcdf=, then is was
not written by libnetcdf -- unless of course some other package is faking it :-)

As an aside, I would encourage the h5netcdf group to start putting its library info in any _NCProperties
attribute that it writes (in place of the netcdf= part of the _NCProperties. perhaps something
like h5netcdf=

[kmuehlbauer]

Yes, you would have to invoke via kwarg invalid_netcdf=True, when opening for create/append.

[dopplershift]

Maybe include that flag in _NCProperties? Though I'd prefer @DennisHeimbigner and @WardF to weigh in here.

[DennisHeimbigner]

The _NCProperties attribute is designed to accept any key=value pairs, so if you want to
add it to your NCProperties, go ahead, It will be ignored by libnetcdf.
We should probably agree on the key name so that is is clear who is producing it:
"h5netcdf_externsions=true", maybe?

[WardF]

note: I realize I used "data model" in a lot of places where I should have said "specification". Substituting this is left as an exercise for the reader 😅

I need to review this thread and discussion to weigh in fully, and I won't be back in the office until Monday. I do want to weigh in with my initial thoughts, however. Forgive me if points I raise are addressed above, I'm on the iOS GitHub client :).

Beyond libnetcdf,netCDF is defined by various data models; libnetcdf implements the features required to read/write these data models. Files that are written using features not included in these data models simply aren't netCDF files, period, regardless of whether libnetcdf can read them, or not.

As we continue to move towards separating the data models from the underlying storage (see ncZarr as the pilot for this effort), we will also need to remain aware of the more holistic "what is netCDF" question posed by Sean Arms last year. A netCDF file defined by a particular data model should have as few data-storage-specific requirements as possible. This will be necessary to obviate the technical debt/burden on the Unidata team/the broader developer community, as well as our associated projects (like netCDF Java).

Ok, that's enough from the phone. Again, forgive me if all of this has been addressed above 😄. While we don't want to fall into the trap of "The Unidata implementation is the only valid implementation", we also need to be careful to avoid "Unidata doesn't have the resources to update libnetcdf to read (what Unidata says) are valid netCDF files".

Put another way, I don't want to say "libnetcdf must be used to write a netCDF file for it to be valid", but if it doesn't adhere to a published data model, then we can't say it's a netCDF file. If it does adhere to the data model/specification but libnetcdf can't read it, then we really have a problem.

[kmuehlbauer]

Thanks all for the discussion so far. I'll recap the current h5netcdf status up to version 0.15.0 status with regard to that issue.

1. General procedure: h5netcdf aims to adhere to NetCDF-4 standard
   1a. writes _NCProperties to newly created files containing eg version=2,h5netcdf=0.15.0,hdf5=1.12.1,h5py=3.4.0
   1b. doesn't change _NCProperties for existing files
   1c. raises CompatibilityError if invalid_netcdf features are about to be written
2. Files are opened with kwargs invalid_netcdf=True for append/create
   2a. doesn't write _NCProperties to newly created files at all
   2b. doesn't change _NCProperties for existing files

We recently identified 2b as a possible bug. So we changed the behavior in a first step for next versions to:

1. General procedure: h5netcdf aims to adhere to NetCDF-4 standard
   1a. writes _NCProperties to newly created files containing eg version=2,h5netcdf=0.15.0,hdf5=1.12.1,h5py=3.4.0
   1b. doesn't change _NCProperties for existing files
   1c. raises CompatibilityError if invalid_netcdf features are about to be written
2. Files are opened with kwargs invalid_netcdf=True for append/create
   2a. doesn't write _NCProperties to newly created files at all
   2b. removes _NCProperties from existing files
   2c. emit UserWarning if the file extension is .nc:
   You are writing invalid netcdf features to file test.nc. The file will thus be not conforming to NetCDF-4 standard and might not be readable by other netcdf tools. Consider using a different extension.

Following the discussion up to now this still can be improved. So we should maybe change 1b. and 2a. ¹ as well as 2b. to also write a specific _NCProperties attribute in those cases (as suggested by @DennisHeimbigner and @dopplershift). I would strongly rely on Unidata and other involved parties expertise to make the changes needed based on the outcome of this discussion.

h5netcdf release 1.0 is planned for next week but this milestone can and should be postponed until coming to a conclusion here. It would be unsatisfying if we move to 1.0 without a proper future-proof accepted solution.

Footnotes

1. "_NCProperties Holds provenance information about a file at the time it was created. It specifies the versions of the netCDF and HDF5 libraries used to create the file." - That quoting from the doc was the main purpose for not changing _NCProperties in existing files. ↩

[DennisHeimbigner]

Yes, that seems appropriate. But does that mean that you do not use libhdf5 at all?
If you do use it, then you should add it to _NCProperties.

[kmuehlbauer]

Sure, h5netcdf does use libhdf5 via h5py. It's the hdf5-part in the above string.

[DennisHeimbigner]

Sorry, I guess I was too cursory.

[DennisHeimbigner]

Ward's comment is an appropriate reminder. In the end, perhaps we (Unidata),
need to be able to say that file X is non-standard, but readable by libnetcdf,
but that we make no guarantee that the file will continue to be readable in the future

[edwardhartnett]

The CF conventions provide an attribute that can be used for this, "history" I believe...