streamsets
Elasticsearch Update Operation Tutorial
Hi,
Is there a tutorial on ElasticSearch update operation?
Pipeline Flow: Have create a minimal pipeline to read data from ES 6.2.2 (origin) and push it to RDBMS (destination 1). Thereafter we update same ES document (destination 2) to update a field @version from 1 to 2. So that on the next run of the pipeline this document from ES will not be picked for moving to RDBMS.
Can someone point to the right direction as we are getting below error, after following the Streamset documents at the Destination (Also if someone could point on debugging the flow it would be helpful as the error is not clear where exactly the issue is) :
2018-05-28 08:27:35,171 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] INFO Pipeline - Destroying pipeline with reason=FAILURE 2018-05-28 08:27:35,177 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] INFO Pipeline - Processing lifecycle stop event 2018-05-28 08:27:35,177 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] INFO Pipeline - Pipeline finished destroying with final reason=FAILURE 2018-05-28 08:27:35,185 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: ELASTICSEARCH_17 - Could not index '347' records: One or more operations failed com.streamsets.pipeline.api.StageException: ELASTICSEARCH_17 - Could not index '347' records: One or more operations failed at com.streamsets.pipeline.stage.destination.elasticsearch.ElasticsearchTarget.write(ElasticsearchTarget.java:342) at com.streamsets.pipeline.configurablestage.DTarget.write(DTarget.java:34) at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:249) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:195) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:257) at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:801) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:846) at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:136) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:845) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:823) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processBatch(ProductionPipelineRunner.java:472) at com.streamsets.datacollector.runner.StageRuntime$3.run(StageRuntime.java:321) at java.security.AccessController.doPrivileged(Native Method) at com.streamsets.datacollector.runner.StageRuntime.processBatch(StageRuntime.java:317) at com.streamsets.datacollector.runner.StageContext.processBatch(StageContext.java:252) at com.streamsets.pipeline.stage.origin.elasticsearch.ElasticsearchSource$ElasticsearchTask.run(ElasticsearchSource.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
