rspatial
Attempt to create 100x 395x87648 TIFF file, but bands must be lesser or equal to 65535
While trying to multiply a multi-layer SpatRaster (with 87648 layers) by an integer number:
imerg.hourly.2006.2010 <- imerg.hourly.2006.2010*0.05
I got the following error message:
Error: [*] failed writing GTiff file
In addition: Warning message:
/dataSLOW/temp4R/spat_jYCcBxnUDThowTR_1425932.tif: Attempt to create 100x
395x87648 TIFF file, but bands must be lesser or equal to 65535. (GDAL er
ror 1)
The file I was using is the following:
imerg.hourly.2001.2005
class : SpatRaster
dimensions : 395, 100, 87648 (nrow, ncol, nlyr)
resolution : 0.1, 0.1 (x, y)
extent : -76, -66, -56.5, -17 (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +a=6378137 +rf=298.257232666016 +no_defs
sources : 3B-HHR-GIS.MS.MRG.3IMERG.20010101-S000000-E002959.0000.V06B.tif
3B-HHR-GIS.MS.MRG.3IMERG.20010101-S003000-E005959.0030.V06B.tif
3B-HHR-GIS.MS.MRG.3IMERG.20010101-S010000-E012959.0060.V06B.tif
... and 87645 more source(s)
names : D2001~00:00, D2001~30:00, D2001~00:00, D2001~30:00, D2001~00:00, D2001~30:00, ...
min values : 0, 0, 0, 0, 0, 0, ...
max values : 194, 203, 204, 195, 189, 189, ...
time : 2001-01-01 00:00:00 to 2005-12-31 23:30:00 GMT
I have 3 questions:
-
Is this a bug or an expected behaviour in some operating systems? (I'm using GNU/Linux)
-
Is there any other way of multiplying a
SpatRasterobject with many layers by a number? -
Currently, it turns out that multiplying a number by a
SpatRasterobject with thousands of layers is a very time consuming operation (hours in some cases). Is there any efficient way of carrying out this multiplication?
Thanks in advance for any help.
mzb
That is a limitation of the GeoTiff or GDAL. The layer (band) is indexed with a small integer, and it cannot be higher than 2^16 = 65536
You can try to write to a NetCDF file using a method that allows setting a filename. For other methods, you can now also set the default filetype to something else like this:
terraOptions(filetype="NetCDF")
I have struggled with performance when dealing with very many layers as well (daily weather data). I have looked at implementing the new GDAL multidimensional dataset but that did not help much. I have some other ideas yet to be explored.
For large number of GTiff layers you may try to set e.g., before loading terra,
Sys.setenv(GDAL_MAX_BAND_COUNT = 1000000)
See https://github.com/r-spatial/stars/issues/467#issuecomment-961876870
Thanks, after loading terra (and on windows) this should then also work
setGDALconfig("GDAL_MAX_BAND_COUNT", "1000000")
Thank you very much for your help.
I can confirm that using Sys.setenv(GDAL_MAX_BAND_COUNT = 1000000) before leading terra did not work on my GNU/Linux machine (Linux Mint 20.03), and raised the same error message:
Error: [*] failed writing GTiff file
In addition: Warning message:
/dataFAST/temp4R/spat_bk0zx6bSqvKTq7y_12815.tif: Attempt to create 100x395x87648 TIFF file, but bands must be lesser or equal to 65535. (GDAL error 1)
Unfortunately, using setGDALconfig("GDAL_MAX_BAND_COUNT", "1000000") after loading terra did not work either in my GNU/Linux machine, and triggered the same error:
Error: [*] failed writing GTiff file
In addition: Warning message:
/dataFAST/temp4R/spat_hijecn9TKp1JVJT_13179.tif: Attempt to create 100x395x87648 TIFF file, but bands must be lesser or equal to 65535. (GDAL error 1)
So, at this moment I'm dividing my SpatRaster objects in smaller ones (i.e., less than 65535 layers)...
Thanks for all your help.
the output of my sessionInfo() is:
sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Linux Mint 20.3
Matrix products: default
BLAS/LAPACK: /opt/intel/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64_lin/libmkl_rt.so
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=es_CL.UTF-8 LC_MESSAGES=en_GB.UTF-8
[7] LC_PAPER=es_CL.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=es_CL.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] terra_1.6-3
loaded via a namespace (and not attached):
[1] zoo_1.8-10 tidyselect_1.1.2 purrr_0.3.4
[4] pbapply_1.5-0 splines_4.2.1 lattice_0.20-45
[7] colorspace_2.0-3 vctrs_0.4.1 generics_0.1.3
[10] viridisLite_0.4.0 spacetime_1.2-8 survival_3.3-1
[13] chron_2.3-57 utf8_1.2.2 rlang_1.0.4
[16] e1071_1.7-11 hexbin_1.28.2 pillar_1.8.0
[19] fitdistrplus_1.1-8 foreign_0.8-82 glue_1.6.2
[22] DBI_1.1.3 sp_1.5-0 RColorBrewer_1.1-3
[25] SPEI_1.7 jpeg_0.1-9 lifecycle_1.0.1
[28] plyr_1.8.7 munsell_0.5.0 gtable_0.3.0
[31] raster_3.5-21 caTools_1.18.2 codetools_0.2-18
[34] latticeExtra_0.6-30 maptools_1.1-4 parallel_4.2.1
[37] class_7.3-20 fansi_1.0.3 lmomco_2.3.7
[40] xts_0.12.1 Rcpp_1.0.9 KernSmooth_2.23-20
[43] classInt_0.4-7 scales_1.2.0 FNN_1.1.3.1
[46] deldir_1.0-6 interp_1.1-3 hydroTSM_0.6-8
[49] automap_1.0-16 ggplot2_3.3.6 png_0.1-7
[52] dplyr_1.0.9 rasterVis_0.51.2 grid_4.2.1
[55] tools_4.2.1 bitops_1.0-7 Lmoments_1.3-1
[58] cli_3.3.0 goftest_1.2-3 magrittr_2.0.3
[61] gstat_2.0-9 proxy_0.4-27 tibble_3.1.8
[64] SCI_1.0-2 pkgconfig_2.0.3 Matrix_1.4-1
[67] MASS_7.3-58.1 assertthat_0.2.1 reshape_0.8.9
[70] R6_2.5.1 intervals_0.15.2 compiler_4.2.1
It's indeed a hard limit in the geotiff code: https://github.com/OSGeo/gdal/blob/master/frmts/gtiff/geotiff.cpp#L16791