jlogstash icon indicating copy to clipboard operation
jlogstash copied to clipboard

outputs为es的时候,丢数据问题

Open fl911 opened this issue 5 years ago • 0 comments

  • 前提是配置consistency参数为true
  • es监听中afterBulk(long arg0, BulkRequest arg1, Throwable arg2)方法:
public void afterBulk(long arg0, BulkRequest arg1, Throwable arg2) {
          logger.error("bulk got exception:", arg2);
                        
          for(ActionRequest request : arg1.requests()){
               addFailedMsg(request);
          }
                        
}

该方法中的addFailedMsg(request)传入的参数为ActionRequest,而 addFailedMsg(Object msg):

public void addFailedMsg(Object msg) {
        if (msg instanceof Map) {
            super.addFailedMsg(msg);
            return;
        }

        throw new IllegalArgumentException("addFailedMsg only accept Map instance");
    }

中实际处理时候做了msg是否为 Map 的判断,所以从这里进去的数据永远不会被添加到 failedMsgQueue,从而导致数据丢失

fl911 avatar Jun 04 '20 06:06 fl911