spline-getting-started
spline-getting-started copied to clipboard
AbsaOSS
unknown resolver null
I'm follwing https://absaoss.github.io/spline/#step-by-step and trying to get understand this. so i was able to get Spline up and running using docker-compose
as per the dicumentation [http://localhost:9090/]( Spline UI). and http://localhost:8080/ is running.
so i tried to with below command
pyspark \
--packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 \
--conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
--conf "spark.spline.producer.url=http://localhost:8080/producer"
In the terminal i was able to see below error
:: problems summary ::
:::: ERRORS
unknown resolver null
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
:: retrieving :: org.apache.spark#spark-submit-parent-1bda6b26-985d-462f-b21b-9eb6a34a5555
confs: [default]
0 artifacts copied, 26 already retrieved (0kB/6ms)
23/02/06 17:15:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
then i tried below command to run small pyspark code
spark-submit \
--packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 \
--conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
--conf "spark.spline.producer.url=http://localhost:8080/producer" \
./schema.py`
the error unknown resolver null was there and but code executed successfully but i can't see any output http://localhost:9090/app/events/list here
any suggestions ??
"unknown resolver null" error is unlikely related to Spline, please double check your Spark installation.
Also try to post a full log from pyspark, so we see more context.
please find the full log herewith for below command
pyspark \
--packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 \
--conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
--conf "spark.spline.producer.url=http://localhost:8080/producer"
(base) d5han@d5han-HP-ProBook-440-G8-Notebook-PC:~/$ pyspark --packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" --conf "spark.spline.producer.url=http://localhost:8080/producer"
Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
23/02/06 19:51:46 WARN Utils: Your hostname, d5han-HP-ProBook-440-G8-Notebook-PC resolves to a loopback address: 127.0.1.1; using 192.168.8.103 instead (on interface wlp2s0)
23/02/06 19:51:46 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
:: loading settings :: url = jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/d5han/.ivy2/cache
The jars for the packages stored in: /home/d5han/.ivy2/jars
za.co.absa.spline.agent.spark#spark-3.0-spline-agent-bundle_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-1ace4748-b1af-4ad6-bee2-9f3744ef62c0;1.0
confs: [default]
found za.co.absa.spline.agent.spark#spark-3.0-spline-agent-bundle_2.12;1.0.2 in central
found za.co.absa.spline.agent.spark#agent-core_2.12;1.0.2 in central
found org.scala-lang#scala-compiler;2.12.10 in local-m2-cache
found org.scala-lang#scala-reflect;2.12.10 in local-m2-cache
found org.scala-lang.modules#scala-xml_2.12;1.0.6 in spark-list
found za.co.absa.commons#commons_2.12;1.3.3 in central
found org.scalaj#scalaj-http_2.12;2.4.1 in central
found io.github.classgraph#classgraph;4.8.116 in central
found org.scala-graph#graph-core_2.12;1.12.5 in central
found com.fasterxml.uuid#java-uuid-generator;4.0.1 in central
found org.slf4j#slf4j-api;1.7.16 in spark-list
found com.lihaoyi#fastparse_2.12;2.3.3 in central
found com.lihaoyi#sourcecode_2.12;0.2.3 in central
found com.lihaoyi#geny_2.12;0.6.10 in central
found org.yaml#snakeyaml;1.33 in local-m2-cache
found commons-configuration#commons-configuration;1.6 in spark-list
found commons-collections#commons-collections;3.2.2 in spark-list
found commons-lang#commons-lang;2.6 in spark-list
found commons-logging#commons-logging;1.1.1 in local-m2-cache
found commons-digester#commons-digester;1.8 in spark-list
found commons-beanutils#commons-beanutils;1.9.4 in local-m2-cache
found commons-beanutils#commons-beanutils-core;1.8.0 in spark-list
found com.github.bigwheel#util-backports_2.12;2.1 in central
found org.json4s#json4s-ext_2.12;3.6.10 in central
found joda-time#joda-time;2.9.3 in spark-list
found org.joda#joda-convert;2.2.1 in central
:: resolution report :: resolve 4507ms :: artifacts dl 7ms
:: modules in use:
com.fasterxml.uuid#java-uuid-generator;4.0.1 from central in [default]
com.github.bigwheel#util-backports_2.12;2.1 from central in [default]
com.lihaoyi#fastparse_2.12;2.3.3 from central in [default]
com.lihaoyi#geny_2.12;0.6.10 from central in [default]
com.lihaoyi#sourcecode_2.12;0.2.3 from central in [default]
commons-beanutils#commons-beanutils;1.9.4 from local-m2-cache in [default]
commons-beanutils#commons-beanutils-core;1.8.0 from spark-list in [default]
commons-collections#commons-collections;3.2.2 from spark-list in [default]
commons-configuration#commons-configuration;1.6 from spark-list in [default]
commons-digester#commons-digester;1.8 from spark-list in [default]
commons-lang#commons-lang;2.6 from spark-list in [default]
commons-logging#commons-logging;1.1.1 from local-m2-cache in [default]
io.github.classgraph#classgraph;4.8.116 from central in [default]
joda-time#joda-time;2.9.3 from spark-list in [default]
org.joda#joda-convert;2.2.1 from central in [default]
org.json4s#json4s-ext_2.12;3.6.10 from central in [default]
org.scala-graph#graph-core_2.12;1.12.5 from central in [default]
org.scala-lang#scala-compiler;2.12.10 from local-m2-cache in [default]
org.scala-lang#scala-reflect;2.12.10 from local-m2-cache in [default]
org.scala-lang.modules#scala-xml_2.12;1.0.6 from spark-list in [default]
org.scalaj#scalaj-http_2.12;2.4.1 from central in [default]
org.slf4j#slf4j-api;1.7.16 from spark-list in [default]
org.yaml#snakeyaml;1.33 from local-m2-cache in [default]
za.co.absa.commons#commons_2.12;1.3.3 from central in [default]
za.co.absa.spline.agent.spark#agent-core_2.12;1.0.2 from central in [default]
za.co.absa.spline.agent.spark#spark-3.0-spline-agent-bundle_2.12;1.0.2 from central in [default]
:: evicted modules:
commons-logging#commons-logging;1.2 by [commons-logging#commons-logging;1.1.1] in [default]
commons-logging#commons-logging;1.1 by [commons-logging#commons-logging;1.1.1] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 28 | 1 | 1 | 2 || 26 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: ERRORS
unknown resolver null
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
:: retrieving :: org.apache.spark#spark-submit-parent-1ace4748-b1af-4ad6-bee2-9f3744ef62c0
confs: [default]
0 artifacts copied, 26 already retrieved (0kB/7ms)
23/02/06 19:51:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/06 19:51:57 WARN AutoDiscoveryPluginRegistry: Plugin za.co.absa.spline.harvester.plugin.embedded.NonPersistentActionsCapturePlugin is disabled in the configuration.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.1.1
/_/
Using Python version 3.7.6 (default, Jan 8 2020 19:59:22)
Spark context Web UI available at http://192.168.8.103:4040
Spark context available as 'sc' (master = local[*], app id = local-1675693313073).
SparkSession available as 'spark'.
>>>
