iotdb icon indicating copy to clipboard operation
iotdb copied to clipboard
verified publisher icon apache

[Bug] MQTT消息插入失败，控制台反复执行

Open yjc980121 opened this issue 2 years ago • 7 comments

Search before asking

  • [X] I searched in the issues and found nothing similar.

Version

iotdb:1.2.1 windows server 2016 openjdk 11

Describe the bug and provide the minimal reproduce step

写入iotdb为mqtt方式，之前有物理量名字不合法，中间有个空格。现在已经修改正确。 控制台情况 Uploading 控制台报错.png…

What did you expect to see?

报错日志如下 Uploading log_datanode_compaction.log…

What did you see instead?

猜测原因应该是mqtt消息可能有缓存，失败了会继续重新执行，但是不合法的消息会永久失败，可以对缓存做一定处理，如几次缓存都插入执行失败后舍弃mqtt消息内容，或者对持续插入失败的消息，单独给一个修订方式等

Anything else?

No response

Are you willing to submit a PR?

  • [ ] I'm willing to submit a PR!

yjc980121 avatar Oct 16 '23 06:10 yjc980121

控制台报错

yjc980121 avatar Oct 16 '23 06:10 yjc980121

log_datanode_compaction.log 报错日志

yjc980121 avatar Oct 16 '23 06:10 yjc980121

2023-10-16 11:23:26,767 [pool-21-IoTDB-Compaction-Worker-10] ERROR o.a.i.d.s.d.c.e.t.InnerSpaceCompactionTask:305 - root.advint-0 [Compaction] Meet errors in inner space compaction. org.apache.iotdb.commons.exception.IllegalPathException: root.advint.zj3j.SiemensPLC1500.停止_桁架 1 is not a legal path at org.apache.iotdb.commons.utils.PathUtils.splitPathToDetachedNodes(PathUtils.java:48) at org.apache.iotdb.commons.path.PartialPath.(PartialPath.java:81) at org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.impl.ReadChunkCompactionPerformer.compactNotAlignedSeries(ReadChunkCompactionPerformer.java:167) at org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.impl.ReadChunkCompactionPerformer.perform(ReadChunkCompactionPerformer.java:85) at org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.InnerSpaceCompactionTask.doCompaction(InnerSpaceCompactionTask.java:175) at org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.AbstractCompactionTask.start(AbstractCompactionTask.java:106) at org.apache.iotdb.db.storageengine.dataregion.compaction.schedule.CompactionWorker.run(CompactionWorker.java:63) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2023-10-16 11:23:26,770 [pool-21-IoTDB-Compaction-Worker-10] INFO o.a.i.d.s.d.c.e.e.CompactionExceptionHandler:65 - root.advint-0 [Compaction][ExceptionHandler] inner space compaction start handling exception, source seqFiles is [file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695859203957-1-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695870501065-2-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695881901479-3-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695893301111-4-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695904700474-5-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695916100457-6-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695927516386-7-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695938901248-8-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695950301304-9-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695961699904-10-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695972501751-11-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695983302066-12-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1695994108408-13-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696005508451-14-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696016908309-15-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696028308468-16-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696039708385-17-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696051101861-18-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696062499280-19-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696073898799-20-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696085298706-21-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696096702272-22-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696108098403-23-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696119498925-24-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696130897635-25-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696141700741-26-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696152503489-27-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696163307206-28-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696174707009-29-0-0.tsfile, status: COMPACTING, file is D:\iotdb\data\datanode\data\sequence\root.advint\0\2804\1696186107015-30-0-0.tsfile, status: COMPACTING], source unseqFiles is [].

yjc980121 avatar Oct 16 '23 06:10 yjc980121

路径修改正确后可以尝试一下清理掉data目录重新写入，如果已经写入了脏数据有可能导致这个问题。

HTHou avatar Oct 16 '23 09:10 HTHou

另外 mqtt的发送端程序也尽量指定一下 utf-8 编码。报错的路径不合法大概率是编码不一致导致的。

HTHou avatar Oct 16 '23 09:10 HTHou

路径修改正确后可以尝试一下清理掉data目录重新写入，如果已经写入了脏数据有可能导致这个问题。

data目录内有数据如何清理？我不能为了解决mqtt错误消息而把整个data数据全删了吧，那我使用iotdb的价值又在哪里？

yjc980121 avatar Oct 19 '23 06:10 yjc980121

另外一个现象，log_confignode_error.log中有如下错误信息，我的数据好长一段时间都没存储到数据库，应该也是和MQTT消息中不合法测量量导致的。我清除data目录后，一切正常。

2023-10-20 07:28:27,062 [pool-14-IoTDB-ConfigNodeRPC-Processor-10] ERROR o.a.t.ProcessFunction:47 - Internal error processing getOrCreateDataPartitionTable java.lang.NullPointerException: null at org.apache.iotdb.confignode.manager.load.balancer.PartitionBalancer.allocateDataPartition(PartitionBalancer.java:193) at org.apache.iotdb.confignode.manager.load.LoadManager.allocateDataPartition(LoadManager.java:134) at org.apache.iotdb.confignode.manager.partition.PartitionManager.getOrCreateDataPartition(PartitionManager.java:374) at org.apache.iotdb.confignode.manager.ConfigManager.getOrCreateDataPartition(ConfigManager.java:824) at org.apache.iotdb.confignode.service.thrift.ConfigNodeRPCServiceProcessor.getOrCreateDataPartitionTable(ConfigNodeRPCServiceProcessor.java:501) at org.apache.iotdb.confignode.rpc.thrift.IConfigNodeRPCService$Processor$getOrCreateDataPartitionTable.getResult(IConfigNodeRPCService.java:7437) at org.apache.iotdb.confignode.rpc.thrift.IConfigNodeRPCService$Processor$getOrCreateDataPartitionTable.getResult(IConfigNodeRPCService.java:7417) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2023-10-20 07:28:28,073 [pool-14-IoTDB-ConfigNodeRPC-Processor-10] ERROR o.a.t.ProcessFunction:47 - Internal error processing getOrCreateDataPartitionTable java.lang.NullPointerException: null at org.apache.iotdb.confignode.manager.load.balancer.PartitionBalancer.allocateDataPartition(PartitionBalancer.java:193) at org.apache.iotdb.confignode.manager.load.LoadManager.allocateDataPartition(LoadManager.java:134) at org.apache.iotdb.confignode.manager.partition.PartitionManager.getOrCreateDataPartition(PartitionManager.java:374) at org.apache.iotdb.confignode.manager.ConfigManager.getOrCreateDataPartition(ConfigManager.java:824) at org.apache.iotdb.confignode.service.thrift.ConfigNodeRPCServiceProcessor.getOrCreateDataPartitionTable(ConfigNodeRPCServiceProcessor.java:501) at org.apache.iotdb.confignode.rpc.thrift.IConfigNodeRPCService$Processor$getOrCreateDataPartitionTable.getResult(IConfigNodeRPCService.java:7437) at org.apache.iotdb.confignode.rpc.thrift.IConfigNodeRPCService$Processor$getOrCreateDataPartitionTable.getResult(IConfigNodeRPCService.java:7417) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

yjc980121 avatar Oct 20 '23 01:10 yjc980121