spring-batch icon indicating copy to clipboard operation
spring-batch copied to clipboard

Improve performance when polling replies in MessageChannelPartitionHandler

Open lcmarvin opened this issue 3 years ago • 0 comments

Currently a for each loop is used in MessageChannelPartitionHandler to poll the partition StepExecutions.

for (Iterator<StepExecution> stepExecutionIterator = split.iterator(); stepExecutionIterator.hasNext();) {
	StepExecution curStepExecution = stepExecutionIterator.next();

	if (!result.contains(curStepExecution)) {
		StepExecution partitionStepExecution = jobExplorer
            .getStepExecution(managerStepExecution.getJobExecutionId(), curStepExecution.getId());

		if (!partitionStepExecution.getStatus().isRunning()) {
			result.add(partitionStepExecution);
		}
	}
}

When there are lots of partition StepExecutions, the method org.springframework.batch.core.explore.JobExplorer#getStepExecution will cause lots of repeated sql query on database. Take a look at the implementation of this method.

	@Nullable
	@Override
	public StepExecution getStepExecution(@Nullable Long jobExecutionId, @Nullable Long executionId) {
		JobExecution jobExecution = jobExecutionDao.getJobExecution(jobExecutionId);
		if (jobExecution == null) {
			return null;
		}
		getJobExecutionDependencies(jobExecution);
		StepExecution stepExecution = stepExecutionDao.getStepExecution(jobExecution, executionId);
		getStepExecutionDependencies(stepExecution);
		return stepExecution;
	}

The JobInstance, JobExecution, JobParameters, JobExecutionContext, and StepExecutions of the JobExecution are queried again and again, which can be simplified in my option.

So can we only query the needed StepExecution in this loop? I believe there will be performance improvement if we can simplify the query.

lcmarvin avatar Jun 17 '22 15:06 lcmarvin