MarquezProject
When I open the dataset, I get "Something went wrong while fetching dataset runs"
Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!
![boring-cyborg[bot] avatar](https://avatars.githubusercontent.com/in/50386?v=4 "Published by a pub.dev verified publisher"){kind=small}
This problem did not arise until the complexity of the data set increased.
The logs is
172.18.0.5 - - [26/Aug/2024:04:02:48 +0000] "GET /api/v1/namespaces/default/datasets?limit=20&offset=0 HTTP/1.1" 200 30 "http://192.168.1.239:3000/datasets" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 27
INFO [2024-08-26 04:02:52,965] marquez.logging.LoggingMdcFilter: status: 200
INFO [2024-08-26 04:02:52,965] marquez.logging.LoggingMdcFilter: status: 200
172.18.0.5 - - [26/Aug/2024:04:02:52 +0000] "GET /api/v1/tags HTTP/1.1" 200 145 "http://192.168.1.239:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 10
172.18.0.5 - - [26/Aug/2024:04:02:52 +0000] "GET /api/v1/namespaces HTTP/1.1" 200 420 "http://192.168.1.239:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 10
INFO [2024-08-26 04:02:53,114] marquez.logging.LoggingMdcFilter: status: 200
172.18.0.5 - - [26/Aug/2024:04:02:53 +0000] "GET /api/v1/namespaces/default/jobs?limit=20&offset=0 HTTP/1.1" 200 11403 "http://192.168.1.239:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 67
INFO [2024-08-26 04:03:12,517] marquez.logging.LoggingMdcFilter: status: 200
172.18.0.5 - - [26/Aug/2024:04:03:12 +0000] "GET /api/v1/namespaces/default/datasets?limit=20&offset=0 HTTP/1.1" 200 30 "http://192.168.1.239:3000/datasets" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 18
ERROR [2024-08-26 04:03:18,847] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: b59ce4fe462660f5
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.
@swaggy111 Thanks for your reply!😃 Only .exe files in the zip file? My marquez is installed in docker on a linux server. How should I use it.
The user appears to be spamming malware unfortunately, please do not click the link. (They had posted the exact same thing in a repository I was working on and searching their username shows a bunch of other issues they've commented this exact link in).
Hey folks I'm facing the same issue :( We updated from 0.47.0 to 0.49.0 and started to have the same error message about java.lang.NullPointerException: namespace is marked non-null but is null when performing a list command on datasets (such as the one done by the Web UI).
@phixMe I'm afraid I don't have it anymore, since we reverted to 0.47.0. I shall try reproducing the error in my test environment and let you know the full stack trace. Thank you!
@phixMe
Hi, I'm encountering the same issue with docker image docker.io/marquezproject/marquez@sha256:da785450d051c079058b87d97ded04148d9ac5e2e8c3fcf4942ddad9cbb1e9ad.
10.0.1.7 - - [03/Oct/2024:07:35:32 +0000] "GET /api/v1/namespaces/jobs_trino_bigs2/datasets?limit=20&offset=0 HTTP/1.1" 200 30 "https://marquez.offuscated.domain.name:3000/datasets" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 14
ERROR [2024-10-03 07:35:37,668] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: c0fc82db2cfd017e
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.<init>(InputFieldNodeData.java:16)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59)
! at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
! at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
! at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
! at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
! at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
! at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
! at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
! at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.toInputFields(ColumnLineageNodeDataMapper.java:60)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.map(ColumnLineageNodeDataMapper.java:41)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.map(ColumnLineageNodeDataMapper.java:28)
! at org.jdbi.v3.core.result.internal.ResultSetResultIterator.next(ResultSetResultIterator.java:100)
! at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
! at org.jdbi.v3.core.result.ResultIterator.forEachRemaining(ResultIterator.java:39)
! at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
! at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
! at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
! at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
! at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
! at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
! at org.jdbi.v3.core.result.ResultIterable.collect(ResultIterable.java:341)
! at org.jdbi.v3.sqlobject.statement.internal.ResultReturner$CollectedResultReturner.mappedResult(ResultReturner.java:293)
! at org.jdbi.v3.sqlobject.statement.internal.SqlQueryHandler.lambda$configureReturner$0(SqlQueryHandler.java:65)
! at org.jdbi.v3.sqlobject.statement.internal.CustomizingStatementHandler.invoke(CustomizingStatementHandler.java:191)
! at org.jdbi.v3.sqlobject.statement.internal.SqlQueryHandler.invoke(SqlQueryHandler.java:27)
! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.lambda$invoke$0(ExtensionMetadata.java:345)
! at org.jdbi.v3.core.AbstractHandleSupplier.invokeInContext(AbstractHandleSupplier.java:36)
! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.call(ExtensionMetadata.java:363)
! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.invoke(ExtensionMetadata.java:346)
! at org.jdbi.v3.core.extension.ExtensionFactoryDelegate.lambda$attach$0(ExtensionFactoryDelegate.java:118)
! at org.jdbi.v3.core.internal.OnDemandExtensions.invoke(OnDemandExtensions.java:85)
! at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$2(OnDemandExtensions.java:71)
! at org.jdbi.v3.core.Jdbi.callWithExtension(Jdbi.java:560)
! at org.jdbi.v3.core.Jdbi.withExtension(Jdbi.java:547)
! at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$3(OnDemandExtensions.java:71)
! at jdk.proxy2/jdk.proxy2.$Proxy73.getLineageRowsForDatasets(Unknown Source)
! at marquez.service.DelegatingDaos$DelegatingColumnLineageDao.getLineageRowsForDatasets(DelegatingDaos.java:99)
! at marquez.service.ColumnLineageService.enrichWithColumnLineage(ColumnLineageService.java:209)
! at marquez.api.DatasetResource.list(DatasetResource.java:156)
! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
! at java.base/java.lang.reflect.Method.invoke(Method.java:569)
! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322)
! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
! at org.eclipse.jetty.server.Server.handle(Server.java:516)
! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
! at java.base/java.lang.Thread.run(Thread.java:840)
ERROR [2024-10-03 07:35:37,673] marquez.logging.LoggingMdcFilter: status: 500
10.0.1.7 - - [03/Oct/2024:07:35:37 +0000] "GET /api/v1/namespaces/trino%3A%2F%2Ftrino1.offuscated.domain.name%3A8443/datasets?limit=20&offset=0 HTTP/1.1" 500 110 "https://marquez.offuscated.domain.name:3000/datasets" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 938
Thanks @wenjiechen-cwj for reporting this issue and @fromcloudbase for confirming.
Looking at both stacktraces,
// @wenjiechen-cwj
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.(InputFieldNodeData.java:16)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59)
// @fromcloudbase
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.<init>(InputFieldNodeData.java:16)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59)
It looks to me that ColumnLineageNodeDataMapper is the cause of NullPointerException. On InputFieldNodeData instantiation, namespace, dataset, and field cannot be null;
We will look deeper into the root cause of the issue and get out a hot fix as soon as we can. @fromcloudbase if you can share your OpenLineage events that you used to get into this state, I'd be very helpful.
Sorry @wslulciuc I've played a moment with 0;49.0, thinking a time problem can be related to Trino's jobs namespace name. But in vain and I've since downgraded to 0.47.0 too. Thanks for your time and efforts.