The CDMDSP ignores ncml and thredds type datasets

[cofinoa]

ncml files (and probably more types) are ignored as netcdf files. The dataset should be opened using the netcdfDataset.open methods.

https://github.com/Unidata/netcdf-java/blob/c74b0bbd3c0d0797c539cf43afff79db34ea4b24/dap4/d4cdm/src/main/java/dap4/cdm/dsp/CDMDSP.java#L117-L118

This relates with issue Unidata/tds#30

N.B: @zequihg50 this FYI

[lesserwhirls]

@DennisHeimbigner - tagging you on this one.

[DennisHeimbigner]

I am aware of the problem, although it dropped off my radar recently. The problem is that it requires some significant re-factoring of the code to work with a NetcdfFIle instead of the string file name and I could not figure out a good solution.

[DennisHeimbigner]

I reviewed my experimental changes that are in another branch. I believe the situation is that the DAP4 code has support for data sources other than those that can be represented as NetcdfFile objects. In particular, it also handles streams and files containing direct DAP4 protocol encoded information.

These are used as follows:

1. Data sources representable as NetcdfFile objects are used by CDMDSP on servers to provide an API that the DAP4 code can then convert to the DAP4 protocol and send out to clients.
2. DAP4 encoded data sources are used by client programs to interpret and make available on clients in the form of CDM. So we have these cases:
3. NetcdfFile -> DSP (i.e. CDMDSP) -> DAP4 (on the server)
4. DAP4 -> DSP (i.e. HttpDSP) -> CDM (on the client)
5. DAP4 test file data -> DSP (i.e. FileDSP) -> DAP4 (on the server)
6. DAP4 test file data -> DSP (i.e. FileDSP) -> CDM (on the client)

The last two cases are specifically for testing, so cases 1 and 2 are the critical ones.

The simplest solution is to provide parallel APIs: one for NetcdfFile data sources (case 1) and one for DAP4 data sources (case 2). The exact API and class structure is TBD.

[cofinoa]

But I think that one of the problems is that CDMDSP it's opening a CDM dataset, first as NetcdfFile and then passing that object to the NetcdfDataset to enhance it as CDM dataset. This way of opening the CDM datasets overlooks the Netcdf dataset open/acquire methods which are able to open datasets defined as NcML, thredds or cdmremote and many more cases. The netcdf file open methods "only" understand files which are understandable under the IOSP API.

Taking into account above, first why the a DAP4 source file can not be open as IOSP to be mapped to the CDM API.

Second question is, why there is a distinction between netcdf datasets and netcdf files. For example, ncml files are a way of encoding a netcdf data product as XML.

Third it's that this DAP4, should be netcdf files or netcdf datasets?

[DennisHeimbigner]

The distinction between NetcdfFile and NetcdfDataset is partly historical. The NetcdfDataset does not require an underlying file whereas NetcdfFile does. Also, If memory serves, NetcdfDataset is the place where coordinate systems are attached. As for IOSP, as a rule, the IOSP is hidden behind NetcdfFile or NetcdfDataset. THe OpenDAP code is an example where NetcdfDataset is used and there is no OpenDAP IOSP. Much of this could benefit from major refactoring, but for now, we are stuck with it.