mosaic
mosaic copied to clipboard
An extension to the Apache Spark framework that allows easy and fast processing of very large geospatial datasets.
I am reading a 1MB snappy parquet file in Databricks within the DataFrame `df`. The following code takes a couple of seconds to terminate: ```python df.withColumn("geometry_27000", st_setsrid(st_geomfromwkt("geometry"), F.lit(27700))) ``` as...
- DBR 13.3 LTS ML (includes Apache Spark 3.4.1, Scala 2.12) - Standard_DS13_v2 (driver and 2 workers) - Photon disabled - databricks-mosaic 0.4.1 - GDAL init script installed in cluster...
Hello. Second file (87mb) works. First (7.9GB) does not. I recall there was an issue with reading files larger than 2GB, but I thought that this had been [resolved with...
The script here says that if UbuntuGIS flag is used, then the GDAL installation will be 3.4.3 https://github.com/databrickslabs/mosaic/blob/main/scripts/mosaic-gdal-init.sh . However UbuntuGIS actually now provides 3.6.4 https://launchpad.net/~ubuntugis/+archive/ubuntu/ppa I think that using...
**Describe the bug** In version 0.4.1; utilizing the function `grid_tessellateexplode()` may inadvertently create topological exceptions which prevent common functions like `display()`, `count()`, `saveAsTable()` from operating and produce an error. This...
Bumps [org.scalamock:scalamock_2.12](https://github.com/paulbutcher/ScalaMock) from 5.2.0 to 6.0.0. Release notes Sourced from org.scalamock:scalamock_2.12's releases. 6.0.0 What's Changed Update sbt to 1.6.1 by @scala-steward in paulbutcher/ScalaMock#428 Update scalatest to 3.2.11 by @scala-steward in...
## Background Our team is using Databricks on GCP and we are dependent on UDFs. As you can see [here](https://docs.gcp.databricks.com/en/udf/unity-catalog.html), they are only available on GCP from DBR14.1 and above...
**Describe the bug** After creating geometries utilizing `st_aswkb()` `st_aswkt()`, some downstream operations may cause an UnknownAuthorityCodeException: epsg:0 error when SRID not specifically declared. This has the effect of preventing certain...