matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[Bug]: 2.1.1-hotfix tke regression: tpch 1t 4cn test 'rpc timeout'

Open Ariznawlll opened this issue 7 months ago • 3 comments

Is there an existing issue for the same bug?

  • [x] I have checked the existing issues.

Branch Name

2.2-dev

Commit ID

34de694e76d46e14f22f51c35be4791054e8fdef

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/15532987124/job/43726944870

Image

log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22Ykg%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-34de694e7-20250609%5C%22%7D%20%7C%3D%20%60rpc%20timeout%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221749529220000%22,%22to%22:%221749529820000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

trigger tke workflow

Additional information

No response

Ariznawlll avatar Jun 10 '25 06:06 Ariznawlll

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22Ykg%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-34de694e7-20250609%5C%22,%20pod%3D%5C%22nightly-regression-dis-tp-cn-jw8wf%5C%22%7D%20%7C%3D%20%60slow%20event%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221749529440000%22,%22to%22:%221749529606000%22%7D%7D%7D&schemaVersion=1&orgId=1 在这个时间段有一个cn s3的读压力很大,延迟很高,导致rpc timeout

LeftHandCold avatar Jun 10 '25 07:06 LeftHandCold

Image

LeftHandCold avatar Jun 10 '25 07:06 LeftHandCold

https://grafana.ci.matrixorigin.cn/d/ae3ttwut16qdca/fileservice-metrics?orgId=1&from=1749529200000&to=1749531599000&var-interval=1m&var-namespace=mo-branch-nightly-34de694e7-20250609&var-pod=nightly-regression-dis-tp-cn-jw8wf

LeftHandCold avatar Jun 10 '25 07:06 LeftHandCold