netcdf4-python can't run in celery/django

Hello， It works ok as normal, But when I run the netcdf in celery task/django,It can't work,and show me this error:

File "...", line 53, in ...
    with Dataset(path, format='NETCDF4') as dataset:
  File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1855, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/my/nc/path'

What happened when I use celery/django?

Jun 10 '19 06:06 wsf1990

I checked the source code, May be the question is at the 2232 line in file netCDF4/_netCDF4.pyx: https://github.com/Unidata/netcdf4-python/blob/master/netCDF4/_netCDF4.pyx#L2232

ierr = nc_open(path, NC_NOWRITE, &grpid)

But I don't know what's wrong with this code.

Jun 10 '19 09:06 wsf1990

Is '/my/nc/path' a real file?

Jun 10 '19 11:06 jswhit

@jswhit Yes, and it can be opened in shell mode

Jun 10 '19 11:06 wsf1990

And I use ubyte.nc which is your test file,It works well.

Jun 10 '19 11:06 wsf1990

A permissions issues then?

Jun 10 '19 11:06 jswhit

Maybe no, the file in same position.Can I transfor my file to you?

Jun 10 '19 11:06 wsf1990

test.zip

Jun 10 '19 11:06 wsf1990

works well in shell mode. Can't work in django.

Jun 10 '19 12:06 wsf1990

Looks like the traceback is coming from MFDataset, which won't work with this file (it can only open NETCDF3 or NETCDF4_CLASSIC formatted files).

Jun 10 '19 14:06 jswhit

@jswhit I had checked both Dataset and MFDataset,It's all show the same error.

Jun 11 '19 01:06 wsf1990

haven't ever used django or celery myself, so I'm afraid I won't be of much help. Somehow the environment in which your are running the script is different than when you run in the shell - perhaps you're picking up a different version of the python module of C libs, or the user that the script is running under doesn't have permission to read the file?

Jun 11 '19 13:06 jswhit

It must be possible, since this project uses netcdf4-python and django:

https://ncdjango.readthedocs.io/en/latest/

Jun 11 '19 13:06 jswhit

Thanks, I'll have a try.

Jun 11 '19 13:06 wsf1990

I'd try set the dir's permission to 777: chmod -R 777 /my/dir/path. And I found that the 'ubyte.nc' and some other's which you provide can work well,but other's can't.

Jun 12 '19 01:06 wsf1990

This project uses both Django, Celery and NetCDF4 Python: https://github.com/seacast/SharkEyes.

You are most likely not handling your environment correctly.

Jun 20 '19 18:06 MiCurry