spline-spark-agent
spline-spark-agent copied to clipboard
AbsaOSS
Spline agent for Apache Spark
This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade io.github.classgraph:classgraph from 4.8.168 to 4.8.170. :information_source: Keep your dependencies up-to-date....
This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade commons-io:commons-io from 2.15.1 to 2.16.0. :information_source: Keep your dependencies up-to-date....
Addressing #705 , * Makes changes to failing tests to perform different behavior based on Spark version * No new functionality actually required to meet compatibility TODO: - [x] Create...
code like this, while spline cannot get `Input Data Source` ```python from pyspark.sql import SparkSession spark = (SparkSession.builder .config('spark.sql.queryExecutionListeners', 'za.co.absa.spline.harvester.listener.SplineQueryExecutionListener') .config('spark.spline.producer.url', 'http://master-1-1:8080/producer') .enableHiveSupport() .getOrCreate() ) def generate_data(): data = [...
I found that spline in pyspark doesn't track some in memory operations like collect, head and toPandas. Operations count and show are tracked as expected. I used spline with bundle-3.2...
there seems to be binary incompatible changes in API of Delta and Spark SQL that Spline core compiled against Spark 2.4 version cannot work with. E.g. RDDPlugin Todo: 1. Check...
I have a statment in a Notebook in Databricks that is ```sql create table lineage_data.lineagedemo.dinner_1 AS SELECT recipe_id, concat(app," + ", main," + ",dessert) AS full_menu FROM lineage_data.lineagedemo.menu ``` the...
Hi @wajda I have a spark code that I got from someone in the organization that has Scala Code They have a Configuration.conf like this If I add in the...
When adding the Spline agent bundle to an AWS Glue Python script (Spark 3.3, Python 3), lineage is produced when using the standard patterns like `df = spark.read.csv(file_path, header=True, inferSchema=True)...
Hi, I have an AWS EMR v6.11 (spark 3.2) where I want to run a spark application via spark-submit, with runtime java 17. My application uses spline agent as a...
