possible issue of maximum filepath length when converting CDF weather datafile to TileDB group

[nkarl]

Hello, I have been tasked with researching TileDB for data storage of a large amount of weather data. The data come in both *.nc and *.cdf format. I am currently working on a simple set of one nc and one cdf files. I was able to convert the *.nc file successfully into a TileDB group. However, I was thrown an exception in the case of the *.cdf file.

After copying the directory to a different location closer to the user's home directory, I was able to convert both successfully. It's not a big deal since I can move location, but I thought you might want to be made aware of this problem.

I created a gist and github repo with steps to replicate the problem:

• gist:

https://gist.github.com/nkarl/e6851dd4d455e47c9211471657edee20

• github repo:

https://github.com/nkarl/tiledb-cf-py-maximum-path-length-conversion-issue.git

Here is the traceback message:

[2022-02-23 17:03:16.037] [Process: 16188] [error] [Global] [TileDB::IO] Error: Failed to convert URI 'file:///C:/Users/KARL/Dev/2022.Spring/423-CAPSTONE/DataLakeAPI/Development/LocalData/implementation_TileDB/ceil_sonde/TEST_DIR/data/sgpsondewnpnC1.b1/20201101.232500.cdf.tdb/array0/__1645664596005_1645664596005_9171b349a7db4cdfbba0ce4b71b9bc2e_10/__fragment_metadata.tdb' to path.
[2022-02-23 17:03:16.037] [Process: 16188] [error] [Global] [TileDB::IO] Error: Failed to convert URI 'file:///C:/Users/KARL/Dev/2022.Spring/423-CAPSTONE/DataLakeAPI/Development/LocalData/implementation_TileDB/ceil_sonde/TEST_DIR/data/sgpsondewnpnC1.b1/20201101.232500.cdf.tdb/array0/__1645664596005_1645664596005_9171b349a7db4cdfbba0ce4b71b9bc2e_10/__fragment_metadata.tdb' to path.
[2022-02-23 17:03:16.038] [Process: 16188] [error] [Global] [TileDB::IO] Error: Cannot write to file 'C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\'; File opening error
[2022-02-23 17:03:16.038] [Process: 16188] [error] [Global] [TileDB::IO] Error: Failed to convert URI 'file:///C:/Users/KARL/Dev/2022.Spring/423-CAPSTONE/DataLakeAPI/Development/LocalData/implementation_TileDB/ceil_sonde/TEST_DIR/data/sgpsondewnpnC1.b1/20201101.232500.cdf.tdb/array0/__1645664596005_1645664596005_9171b349a7db4cdfbba0ce4b71b9bc2e_10/__fragment_metadata.tdb' to path.
[2022-02-23 17:03:16.038] [Process: 16188] [error] [Global] [TileDB::IO] Error: Failed to delete file 'C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\'
[2022-02-23 17:03:16.038] [Process: 16188] [error] [Context: 2] Cannot unlock array exclusive lock; Filelock not found
Traceback (most recent call last):
  File "C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\new_ceil_sonde.py", line 100, in <module>
    sonde_converter.convert_to_group(sonde_group_uri)
  File "C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\.virtualenv\lib\site-packages\tiledb\cf\netcdf_engine\converter.py", line 753, in convert_to_group
    self.copy_to_group(
  File "C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\.virtualenv\lib\site-packages\tiledb\cf\netcdf_engine\converter.py", line 924, in copy_to_group
    self._copy_data(
  File "C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\.virtualenv\lib\site-packages\tiledb\cf\netcdf_engine\converter.py", line 461, in _copy_data
    array_creator.copy(
  File "C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\.virtualenv\lib\site-packages\tiledb\cf\netcdf_engine\_array_converters.py", line 199, in copy
    self._copy_to_array(
  File "C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TE

ST_DIR\.virtualenv\lib\site-packages\tiledb\cf\netcdf_engine\_array_converters.py", line 86, in _copy_to_array
    tiledb_array[coord_values] = data
  File "tiledb\libtiledb.pyx", line 5343, in tiledb.libtiledb.DenseArrayImpl.__setitem__
  File "tiledb\libtiledb.pyx", line 5423, in tiledb.libtiledb.DenseArrayImpl._setitem_impl
  File "tiledb\libtiledb.pyx", line 478, in tiledb.libtiledb._write_array
  File "tiledb\libtiledb.pyx", line 542, in tiledb.libtiledb._raise_ctx_err
  File "tiledb\libtiledb.pyx", line 527, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: [TileDB::IO] Error: Cannot write to file 'C:\Users\KARL\Dev\2022.Spring\423-CAPSTONE\DataLakeAPI\Development\LocalData\implementation_TileDB\ceil_sonde\TEST_DIR\'; File opening error

[jp-dark]

Thank you opening up this issue and providing the example, especially since you were able to get your code working anyway. This is related to issue 2627 in the TileDB Embedded library, and I will leave this open until it is resolved there.