Unidata
[5.x]: Problem opening NC4/HDF5 file due to data object header magic
Versions impacted by the bug
v5.x, v6.x, v7.x
What went wrong?
A user developing a data product for ESA was unable to open an NC4/HDF5 data in Panoply (currently based on netcdf-Java 5.4.2) due to an "DataObject doesnt start with OHDR" exception thrown by the H5objects class. The same problem occurs using toolsUI 5.4.2 and the most recent IDV download. (FWIW, the file does open in HDFview, but meh.)
I added some diagnostics to H5objects, and found that the object header that it doesn't like is not "OHDR" but instead the 4-char string "ëHDF". A search of the file using a hex editor indicates that the only place that the latter string occurs is at the very start of the file, and is of course the HDF magic string. So it seems that somehow when NJ tries to read the data object, an offset of 0 is being applied?
So although I was originally prone to suggest, "Your file is messed up", I am instead asking if instead the file is merely quirky and it is somehow triggering a bug in netCDF-Java. Unfortunately, this is deep enough voodoo that I am posting to the Github repo for help rather than spend a couple weeks trying to parse the file vis a vis HDF format.
A copy of the problematic file is temporarily here on Dropbox. The dataset variable apparently involved with the exception is ricfer_channel_name -- omission of that variable from the dataset results in apps being able to open it.
Thx.
Relevant stack trace
No response
Relevant log messages
No response
If you have an example file that you can share, please attach it to this issue.
If so, may we include it in our test datasets to help ensure the bug does not return once fixed? Note: the test datasets are publicly accessible without restriction.
Yes
Code of Conduct
- [X] I agree to follow the UCAR/Unidata Code of Conduct
Hmmm...this is indeed an interesting case. The root issue is netCDF-Java is struggling with the DIMENSION_LIST attribute of the ricfer_channel_name dataset (full path /Annotation data/Instrumental information/ricfer_channel_name). The DIMENSION_LIST holds a reference, and we are reading the heap address of the reference as 0, which isn't correct. Interestingly enough, if I copy the dataset using nccopy (nccopy -k nc4 FLX_GPP__BRK_L1A____20190914T103613_20190914T103615_20210910T112811__18260.HRE1.hdf rewrite.hdf), things work. It makes me wonder if there was a bug in netCDF-C or HDF5 (netcdflibversion=4.6.0|hdf5libversion=1.10.0) that was addressed by rewriting the file (netcdf=4.7.3,hdf5=1.10.4), and perhaps the netCDF-C library has a workaround in place that allows things to work? @JohnLCaron - have you seen behavior like this before?
I saw the same thing as you - a zero as a reference in a DIMENSION_LIST object. I dont remember this specific thing, or a workaround.
It would take me a while to spin up more on it, which I wont be able to do right now. If nccopy repairs it, its certainly an indication of a bug in the original hdf5 library, now fixed. Or a lucky coincidence.
On Tue, Sep 14, 2021 at 9:55 AM Sean Arms @.***> wrote:
Hmmm...this is indeed an interesting case. The root issue is netCDF-Java is struggling with the DIMENSION_LIST attribute of the ricfer_channel_name dataset (full path /Annotation data/Instrumental information/ricfer_channel_name). The DIMENSION_LIST holds a reference, and we are reading the heap address of the reference as 0, which isn't correct. Interestingly enough, if I copy the dataset using nccopy (nccopy -k nc4 FLX_GPP__BRK_L1A____20190914T103613_20190914T103615_20210910T112811__18260.HRE1.hdf rewrite.hdf), things work. It makes me wonder if there was a bug in netCDF-C or HDF5 (netcdflibversion=4.6.0|hdf5libversion=1.10.0) that was addressed by rewriting the file (netcdf=4.7.3,hdf5=1.10.4), and perhaps the netCDF-C library has a workaround in place that allows things to work? @JohnLCaron https://github.com/JohnLCaron - have you seen behavior like this before?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Unidata/netcdf-java/issues/823#issuecomment-919285562, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVZQHE6VGT4BFGDPZT4NDUB5WAFANCNFSM5D7K64HQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You could check with hdf5 support..
On Wed, Sep 15, 2021 at 12:21 PM John Caron @.***> wrote:
I saw the same thing as you - a zero as a reference in a DIMENSION_LIST object. I dont remember this specific thing, or a workaround.
It would take me a while to spin up more on it, which I wont be able to do right now. If nccopy repairs it, its certainly an indication of a bug in the original hdf5 library, now fixed. Or a lucky coincidence.
On Tue, Sep 14, 2021 at 9:55 AM Sean Arms @.***> wrote:
Hmmm...this is indeed an interesting case. The root issue is netCDF-Java is struggling with the DIMENSION_LIST attribute of the ricfer_channel_name dataset (full path /Annotation data/Instrumental information/ricfer_channel_name). The DIMENSION_LIST holds a reference, and we are reading the heap address of the reference as 0, which isn't correct. Interestingly enough, if I copy the dataset using nccopy (nccopy -k nc4 FLX_GPP__BRK_L1A____20190914T103613_20190914T103615_20210910T112811__18260.HRE1.hdf rewrite.hdf), things work. It makes me wonder if there was a bug in netCDF-C or HDF5 (netcdflibversion=4.6.0|hdf5libversion=1.10.0) that was addressed by rewriting the file (netcdf=4.7.3,hdf5=1.10.4), and perhaps the netCDF-C library has a workaround in place that allows things to work? @JohnLCaron https://github.com/JohnLCaron - have you seen behavior like this before?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Unidata/netcdf-java/issues/823#issuecomment-919285562, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVZQHE6VGT4BFGDPZT4NDUB5WAFANCNFSM5D7K64HQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.