doris icon indicating copy to clipboard operation
doris copied to clipboard

[Fix](Stmt)Move reset insert timeout in handleInsert

Open SaintBacchus opened this issue 2 years ago • 22 comments

Proposed changes

Issue Number: close #xxx

Problem summary

PR #16343 seprate the query_timeout and insert_timeout. But it only change the timeout in ConnectProcessor::handleQuery function which only handle mysql client sql. The CTAS and MTMV insert job will miss these sesssion variable. So I move it into StmtExecutor::handleInsert funtion since all these three feature will go through these code.

Checklist(Required)

  • [ ] Does it affect the original behavior
  • [ ] Has unit tests been added
  • [ ] Has document been added or modified
  • [ ] Does it need to update dependencies
  • [ ] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

SaintBacchus avatar Feb 28 '23 12:02 SaintBacchus

run fe-ut

SaintBacchus avatar Feb 28 '23 12:02 SaintBacchus

run buildall

SaintBacchus avatar Mar 01 '23 01:03 SaintBacchus

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.38 seconds stream load tsv: 474 seconds loaded 74807831229 Bytes, about 150 MB/s stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230302061932_clickbench_pr_106929.html

hello-stephen avatar Mar 01 '23 02:03 hello-stephen

Hi @SaintBacchus , just for disscussion, how about replace the ConnectContext#resetExecTimeout by

public void setExecTimeout(int timeout) {
    executionTimeoutS = timeout;
}

And in StmtExecutor#handleInsertStmt, use

context.setExecTimeout(Math.max(context.getSessionVariable().getInsertTimeoutS(), context.getExecTimeoutS()));

I think it may seem clearer.

Hastyshell avatar Mar 01 '23 04:03 Hastyshell

@TangSiyang2001 thanks for suggestions. I had modified it.

And I place the Math.max in the function setExecTimeout, how do you think about these?

    public void setExecTimeout(int timeout) {
        executionTimeoutS = Math.max(timeout, executionTimeoutS);
    }

SaintBacchus avatar Mar 01 '23 06:03 SaintBacchus

run buildall

SaintBacchus avatar Mar 01 '23 06:03 SaintBacchus

run p1

SaintBacchus avatar Mar 01 '23 09:03 SaintBacchus

run nereids

SaintBacchus avatar Mar 01 '23 09:03 SaintBacchus

run p1

SaintBacchus avatar Mar 01 '23 11:03 SaintBacchus

@TangSiyang2001 thanks for suggestions. I had modified it.

And I place the Math.max in the function setExecTimeout, how do you think about these?

    public void setExecTimeout(int timeout) {
        executionTimeoutS = Math.max(timeout, executionTimeoutS);
    }

Thank u for adopting my idea. I just curious about that if we support a finer-grained timeout in the future, and some stmt will require a shorter timeout than the default exec_timeout(currently query_timeout), in that case max policy in setExecTimeout may not be suitable anymore, and dragging max to the certain stmt, just keep setExecTimeout a simple set operation will be more flexible and extensible for supporting different timeout policy. How do you think :)

Hastyshell avatar Mar 01 '23 11:03 Hastyshell

fine @TangSiyang2001

SaintBacchus avatar Mar 01 '23 11:03 SaintBacchus

run buildall

SaintBacchus avatar Mar 01 '23 11:03 SaintBacchus

PR approved by anyone and no changes requested.

github-actions[bot] avatar Mar 01 '23 12:03 github-actions[bot]

@TangSiyang2001 can you reviewer again? I find that the open txn also need set this timeout

SaintBacchus avatar Mar 02 '23 02:03 SaintBacchus

run p1

SaintBacchus avatar Mar 02 '23 02:03 SaintBacchus

I think use resetExecTimeForInsert to do these job is better than setExecTime since it had called three times.

SaintBacchus avatar Mar 02 '23 02:03 SaintBacchus

run buildall

SaintBacchus avatar Mar 02 '23 02:03 SaintBacchus

run p1

SaintBacchus avatar Mar 02 '23 02:03 SaintBacchus

I think use resetExecTimeForInsert to do these job is better than setExecTime since it had called three times.

Sorry for late reply, you may look at my comment above. Will it be clearer than the former ways?

Hastyshell avatar Mar 02 '23 04:03 Hastyshell

run buildall

SaintBacchus avatar Mar 02 '23 05:03 SaintBacchus

run arm

SaintBacchus avatar Mar 02 '23 09:03 SaintBacchus

run arm

SaintBacchus avatar Mar 02 '23 14:03 SaintBacchus

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Mar 03 '23 02:03 github-actions[bot]