druid icon indicating copy to clipboard operation
druid copied to clipboard
verified publisher icon apache

Cannot create directory /druid/deepstorage

Open renanxx1 opened this issue 3 years ago • 0 comments

I'm facing problems when I try to ingest the example data of the druid, I'm getting permissions error when it tries to create the folder /druid.

I'm running the druid-operator in a local kubernetes cluster using kind.

The deploy was made following the commands bellow.

helm upgrade --install cluster-druid-operator ./chart/ kubectl apply -f ./examples/tiny-cluster-zk.yaml kubectl apply -f ./examples/tiny-cluster.yaml

This is the log I receive when I try to load the example data.

	at org.apache.druid.segment.loading.LocalDataSegmentPusher.pushToPath(LocalDataSegmentPusher.java:92) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.push(LocalDataSegmentPusher.java:68) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:949) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:945) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:779) ~[druid-server-0.22.1.jar:0.22.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2022-07-13T21:53:28,710 WARN [[index_parallel_wikipedia_ickecnkh_2022-07-13T21:53:12.054Z]-appenderator-merge] org.apache.druid.java.util.common.RetryUtils - Retrying (3 of 4) in 4,155ms.
java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/3d6bf7db-b6f1-4bf0-8b9b-392dfdec542b'.
	at org.apache.commons.io.FileUtils.mkdirs(FileUtils.java:2200) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:1383) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.pushToPath(LocalDataSegmentPusher.java:92) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.push(LocalDataSegmentPusher.java:68) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:949) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:945) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:779) ~[druid-server-0.22.1.jar:0.22.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2022-07-13T21:53:32,867 WARN [[index_parallel_wikipedia_ickecnkh_2022-07-13T21:53:12.054Z]-appenderator-merge] org.apache.druid.java.util.common.RetryUtils - Retrying (4 of 4) in 9,969ms.
java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/3a6db598-92d6-4f16-9625-0edaefe16ca9'.
	at org.apache.commons.io.FileUtils.mkdirs(FileUtils.java:2200) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:1383) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.pushToPath(LocalDataSegmentPusher.java:92) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.push(LocalDataSegmentPusher.java:68) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:949) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:945) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:779) ~[druid-server-0.22.1.jar:0.22.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2022-07-13T21:53:42,841 WARN [[index_parallel_wikipedia_ickecnkh_2022-07-13T21:53:12.054Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to push merged index for segment[wikipedia_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2022-07-13T21:53:20.560Z].
java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/42f4296e-c5ef-46f7-a1a0-4ebf3984fd34'.
	at org.apache.commons.io.FileUtils.mkdirs(FileUtils.java:2200) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:1383) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.pushToPath(LocalDataSegmentPusher.java:92) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.push(LocalDataSegmentPusher.java:68) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:949) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:945) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:779) ~[druid-server-0.22.1.jar:0.22.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2022-07-13T21:53:42,845 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/42f4296e-c5ef-46f7-a1a0-4ebf3984fd34'.
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:964) ~[druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:500) [druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:159) [druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1086) [druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:483) [druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:159) [druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:471) [druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:443) [druid-indexing-service-0.22.1.jar:0.22.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/42f4296e-c5ef-46f7-a1a0-4ebf3984fd34'.
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:148) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:132) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:130) ~[druid-indexing-service-0.22.1.jar:0.22.1]
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:891) ~[druid-indexing-service-0.22.1.jar:0.22.1]
	... 11 more
Caused by: java.lang.RuntimeException: java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/42f4296e-c5ef-46f7-a1a0-4ebf3984fd34'.
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:989) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:779) ~[druid-server-0.22.1.jar:0.22.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
	... 3 more
Caused by: java.io.IOException: Cannot create directory '/druid/deepstorage/intermediate_pushes/42f4296e-c5ef-46f7-a1a0-4ebf3984fd34'.
	at org.apache.commons.io.FileUtils.mkdirs(FileUtils.java:2200) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:1383) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.pushToPath(LocalDataSegmentPusher.java:92) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.loading.LocalDataSegmentPusher.push(LocalDataSegmentPusher.java:68) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:949) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:945) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:779) ~[druid-server-0.22.1.jar:0.22.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
	... 3 more
2022-07-13T21:53:42,861 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - handler[index_parallel_wikipedia_ickecnkh_2022-07-13T21:53:12.054Z] not currently registered, ignoring.
2022-07-13T21:53:42,865 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_parallel_wikipedia_ickecnkh_2022-07-13T21:53:12.054Z",
  "status" : "FAILED",
  "duration" : 24139,
  "errorMsg" : "java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: jav...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2022-07-13T21:53:42,871 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2022-07-13T21:53:42,873 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2022-07-13T21:53:42,880 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@7236518{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2022-07-13T21:53:42,880 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2022-07-13T21:53:42,881 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@194b9783{/,null,STOPPED}
2022-07-13T21:53:42,887 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2022-07-13T21:53:42,888 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_parallel_wikipedia_ickecnkh_2022-07-13T21:53:12.054Z].
2022-07-13T21:53:42,906 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2022-07-13T21:53:42,923 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2022-07-13T21:53:43,032 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10005ff8f680005
2022-07-13T21:53:43,032 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x10005ff8f680005 closed
2022-07-13T21:53:43,058 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task

renanxx1 avatar Jul 13 '22 22:07 renanxx1