iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard
verified publisher icon apache

reset catalog can't reslove s3 domain name

Open ming12713 opened this issue 2 years ago • 7 comments

Apache Iceberg version

1.2.1 (latest release)

Query engine

Spark

Please describe the bug 🐞

Follow this bkm test rest catalog is ok

Attempting to modify spark and restcata to with production MinIO address <los.xxxx.com>, there is an issue with can't resolve dns "bucket.los.xxxx.com" Screenshot from 2023-05-26 15-41-30

rest cataserver log

org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=500, type=SdkClientException, message=Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.)
software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59)
	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:202)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
	at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9321)
	at org.apache.iceberg.aws.s3.S3OutputStream.completeUploads(S3OutputStream.java:435)
	at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:269)
	at java.base/sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:347)
	at java.base/sun.nio.cs.StreamEncoder.close(StreamEncoder.java:169)
	at java.base/java.io.OutputStreamWriter.close(OutputStreamWriter.java:252)
	at org.apache.iceberg.TableMetadataParser.$closeResource(TableMetadataParser.java:131)
	at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:131)
	at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:114)
	at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
	at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
	at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:106)
	at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
	at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:199)
	at org.apache.iceberg.rest.CatalogHandlers.createTable(CatalogHandlers.java:214)
	at org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:319)
	at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:384)
	at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
	at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: xx.los.xxxx.com
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	... 59 more
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: xx.los.xxxx.com
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: xx.los.xxxx.com
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: xx.los.xxxx.com
Caused by: java.net.UnknownHostException: xx.los.xxxx.com

However, using IP resolution instead of the domain name does not encounter any problems.

spark conf

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.catalog-impl    org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.demo.uri             http://rest:8181
spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse       s3a://xx/
spark.sql.catalog.demo.s3.endpoint     http://los.xxxx.com:9000
spark.sql.defaultCatalog               demo
spark.eventLog.enabled                 true
spark.eventLog.dir                     /home/iceberg/spark-events
spark.history.fs.logDirectory          /home/iceberg/spark-events
spark.sql.catalogImplementation        in-memory

ming12713 avatar May 26 '23 07:05 ming12713

Same here! It seems that within the Spark context, a different DNS resolver is used. We currently did an ugly workaround by giving the container a fixed IP and reference this in the config.

Any clean solve is much appreciated!

SvenThies avatar May 30 '23 17:05 SvenThies

Same here! It seems that within the Spark context, a different DNS resolver is used. We currently did an ugly workaround by giving the container a fixed IP and reference this in the config.

Any clean solve is much appreciated!

i also use ip as workaroud solution

ming12713 avatar May 31 '23 02:05 ming12713

Can you try setting:

s3.path-style-access=true

This can be done on a catalog level:

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.catalog-impl    org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.demo.uri             http://rest:8181
spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse       s3a://xx/
spark.sql.catalog.demo.s3.endpoint     http://los.xxxx.com:9000
spark.sql.catalog.demo.s3.path-style-access true
spark.sql.defaultCatalog               demo
spark.eventLog.enabled                 true
spark.eventLog.dir                     /home/iceberg/spark-events
spark.history.fs.logDirectory          /home/iceberg/spark-events
spark.sql.catalogImplementation        in-memory

Fokko avatar May 31 '23 15:05 Fokko

Hey @Fokko,

thanks for the suggestions. Unfortunately, it doesn't work. We tried the following

spark.sql.extensions                        org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.default                   org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.default.catalog-impl      org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.default.uri               http://rest:8181
spark.sql.catalog.default.io-impl           org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.default.warehouse         s3://dev-customer-analytics/
spark.sql.catalog.default.s3.endpoint       http://minio:9000
spark.sql.catalog.default.s3.path-style-access true
spark.sql.defaultCatalog                    default
spark.sql.sources.partitionOverwriteMode    dynamic
spark.eventLog.enabled                      true
spark.eventLog.dir                          /home/iceberg/spark-events
spark.history.fs.logDirectory               /home/iceberg/spark-events
spark.sql.catalogImplementation             in-memory
spark.jars.packages                         org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.2,org.apache.kafka:kafka-clients:2.8.1,org.apache.commons:commons-pool2:2.11.1,org.apache.spark:spark-token-provider-kafka-0-10_2.12:3.3.2,com.google.code.findbugs:jsr305:3.0.0,org.apache.spark:spark-tags_2.12:3.3.2

Additionally, we tried setting spark.hadoop.fs.s3a.path.style.access true in the spark-defaults.conf, but didn't work.

SvenThies avatar Jun 01 '23 07:06 SvenThies

Can you try setting:

s3.path-style-access=true

This can be done on a catalog level:

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.catalog-impl    org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.demo.uri             http://rest:8181
spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse       s3a://xx/
spark.sql.catalog.demo.s3.endpoint     http://los.xxxx.com:9000
spark.sql.catalog.demo.s3.path-style-access true
spark.sql.defaultCatalog               demo
spark.eventLog.enabled                 true
spark.eventLog.dir                     /home/iceberg/spark-events
spark.history.fs.logDirectory          /home/iceberg/spark-events
spark.sql.catalogImplementation        in-memory

Thanks @Fokko But, Failed to modify sql.catalog

spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
spark.hadoop.fs.s3a.impl  org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint.region cn-north-1
spark.hadoop.fs.s3a.access.key  xm2U3ww
spark.hadoop.fs.s3a.secret.key  eDMZQAulRQJm
spark.hadoop.fs.s3a.endpoint    http://los.xxx.com
spark.hadoop.fs.s3a.bucket.probe  0
spark.hadoop.fs.s3a.change.detection.version.required  false
spark.hadoop.fs.s3a.change.detection.mode  none
spark.hadoop.fs.s3a.path.style.access  true

# spark jdbc
spark.sql.extensions                      org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions 
spark.sql.catalog.jdbc                    org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.jdbc.catalog-impl       org.apache.iceberg.jdbc.JdbcCatalog 
spark.sql.catalog.jdbc.uri                jdbc:postgresql://10.0.162.5:31884/jdbc_gm
spark.sql.catalog.jdbc.jdbc.user          postgres
spark.sql.catalog.jdbc.jdbc.password      QzBodddd
spark.sql.catalog.jdbc.warehouse          s3://minio-jdbc
spark.sql.catalog.jdbc.io-impl            org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.jdbc.s3.endpoint        http://los.xxx.com
spark.sql.catalog.jdbc.s3.path-style-access  true
spark.eventLog.enabled                    true

errors output Screenshot from 2023-06-06 11-25-34

AWS profile is configured in os profile

ming12713 avatar Jun 06 '23 03:06 ming12713

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

github-actions[bot] avatar Dec 04 '23 00:12 github-actions[bot]

Is there anyone who solved this issue?

dongsupkim-onepredict avatar Feb 27 '24 04:02 dongsupkim-onepredict

Is there anyone who solved this issue?

Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true

It'll be converted to s3.path-style-access=true when the catalog is created and passed to S3FileIO

sg-c0de avatar Mar 13 '24 08:03 sg-c0de

It works Thanks! @sg-c0de

dongsupkim-onepredict avatar Mar 20 '24 04:03 dongsupkim-onepredict

Is there anyone who solved this issue?

Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true

It'll be converted to s3.path-style-access=true when the catalog is created and passed to S3FileIO @sg-c0de Can the jdbc catalog be configured like this?

ming12713 avatar Mar 20 '24 05:03 ming12713

@sg-c0de Thanks. Saved my day. I can't find it in the iceberg document though.

springbearpro avatar Mar 31 '24 17:03 springbearpro

Is there anyone who solved this issue?

Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true It'll be converted to s3.path-style-access=true when the catalog is created and passed to S3FileIO @sg-c0de Can the jdbc catalog be configured like this?

@springbearpro Sure, you can do it by passing this parameter in the config of ETL tool you use. For spark in spark-defaults.conf it will be like: spark.sql.catalog.catalof_name.s3.path-style-access true

Full listing

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.type            jdbc
spark.sql.catalog.demo.uri             jdbc:postgresql://192.168.0.16:5432/jdbc_catalog_test
spark.sql.catalog.demo.jdbc.verifyServerCertificate     false
spark.sql.catalog.demo.jdbc.useSSL      false
spark.sql.catalog.demo.jdbc.user        jdbc
spark.sql.catalog.demo.jdbc.password    pass
spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse       s3://iceberg-test-jdbc/
spark.sql.catalog.demo.s3.endpoint     https://minio.example.com
spark.sql.catalog.demo.s3.path-style-access     true
spark.sql.defaultCatalog               demo
spark.eventLog.enabled                 true
spark.eventLog.dir                     /home/iceberg/spark-events
spark.history.fs.logDirectory          /home/iceberg/spark-events
spark.sql.catalogImplementation        in-memory

For trino it will be s3.path-style-access in the catalog properties https://trino.io/docs/current/object-storage/file-system-s3.html

sg-c0de avatar Mar 31 '24 17:03 sg-c0de

Is there anyone who solved this issue?

Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true It'll be converted to s3.path-style-access=true when the catalog is created and passed to S3FileIO @sg-c0de Can the jdbc catalog be configured like this?

@ming12713 Sure, you can do it by passing this parameter in the config of ETL tool you use. For spark in spark-defaults.conf it will be like: spark.sql.catalog.catalof_name.s3.path-style-access true

Full listing

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.type            jdbc
spark.sql.catalog.demo.uri             jdbc:postgresql://192.168.0.16:5432/jdbc_catalog_test
spark.sql.catalog.demo.jdbc.verifyServerCertificate     false
spark.sql.catalog.demo.jdbc.useSSL      false
spark.sql.catalog.demo.jdbc.user        jdbc
spark.sql.catalog.demo.jdbc.password    pass
spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse       s3://iceberg-test-jdbc/
spark.sql.catalog.demo.s3.endpoint     https://minio.example.com
spark.sql.catalog.demo.s3.path-style-access     true
spark.sql.defaultCatalog               demo
spark.eventLog.enabled                 true
spark.eventLog.dir                     /home/iceberg/spark-events
spark.history.fs.logDirectory          /home/iceberg/spark-events
spark.sql.catalogImplementation        in-memory

For trino it will be s3.path-style-access in the catalog properties https://trino.io/docs/current/object-storage/file-system-s3.html

sg-c0de avatar Apr 01 '24 05:04 sg-c0de

Is there anyone who solved this issue?

Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true It'll be converted to s3.path-style-access=true when the catalog is created and passed to S3FileIO @sg-c0de Can the jdbc catalog be configured like this?

@ming12713 Sure, you can do it by passing this parameter in the config of ETL tool you use. For spark in spark-defaults.conf it will be like: spark.sql.catalog.catalof_name.s3.path-style-access true

Full listing

spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.demo.type            jdbc
spark.sql.catalog.demo.uri             jdbc:postgresql://192.168.0.16:5432/jdbc_catalog_test
spark.sql.catalog.demo.jdbc.verifyServerCertificate     false
spark.sql.catalog.demo.jdbc.useSSL      false
spark.sql.catalog.demo.jdbc.user        jdbc
spark.sql.catalog.demo.jdbc.password    pass
spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.demo.warehouse       s3://iceberg-test-jdbc/
spark.sql.catalog.demo.s3.endpoint     https://minio.example.com
spark.sql.catalog.demo.s3.path-style-access     true
spark.sql.defaultCatalog               demo
spark.eventLog.enabled                 true
spark.eventLog.dir                     /home/iceberg/spark-events
spark.history.fs.logDirectory          /home/iceberg/spark-events
spark.sql.catalogImplementation        in-memory

For trino it will be s3.path-style-access in the catalog properties https://trino.io/docs/current/object-storage/file-system-s3.html

@sg-c0de it's work thanks

ming12713 avatar Apr 01 '24 06:04 ming12713

@sg-c0de Thx, You save my time

inkkim avatar Jul 19 '24 06:07 inkkim