matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard
verified publisher icon matrixorigin

[Tech Request]: Umbrella issue of memory optimization

Open sukki37 opened this issue 2 years ago • 65 comments

Is there an existing issue for enhancement?

  • [ ] I have checked the existing issues.

What would you like to be added?

We experienced many out-of-memory errors, leading to system instability. Memory management needs optimization for preventing OOM scenarios and for better control over memory usage. This issue is a summary of related bug and task issues:

  • [x] #10509
  • [x] #10945
  • [x] #11225
  • [x] #11221
  • [x] #10659
  • [x] #10212
  • [ ] #7505
  • [x] #9008
  • [ ] #11509
  • [ ] https://github.com/matrixorigin/MO-Cloud/issues/984
  • [ ] https://github.com/matrixorigin/MO-Cloud/issues/952
  • [ ] https://github.com/matrixorigin/MO-Cloud/issues/952
  • [x] #10945
  • [x] #12204
  • [x] #12193
  • [ ] https://github.com/matrixorigin/MO-Cloud/issues/1584
  • [ ] #11529

Dependencies:

  • [x] #10370
  • [x] #11511
  • [ ] #11650
  • [x] #11651
  • [x] #11652
  • [ ] #3433
  • [x] #11658
  • [x] #11865
  • [x] #11578
  • [x] #10212

Why is this needed?

No response

Additional information

No response

sukki37 avatar Aug 30 '23 02:08 sukki37

处理daily run的问题中。。还未处理

nnsgmsone avatar Sep 08 '23 11:09 nnsgmsone

讨论data race以及pipeline重构的一些事情，尚未处理

nnsgmsone avatar Sep 11 '23 10:09 nnsgmsone

编写pipeline重构文档中，还未处理。

nnsgmsone avatar Sep 12 '23 11:09 nnsgmsone

编写pipeline重构文档中，还未处理。

nnsgmsone avatar Sep 13 '23 10:09 nnsgmsone

思考pipeline重构的设计中，还未处理

nnsgmsone avatar Sep 14 '23 10:09 nnsgmsone

继续整理pipeline的文档中，还未处理

nnsgmsone avatar Sep 15 '23 10:09 nnsgmsone

明天提交第一个pr，用于管理session的内存

nnsgmsone avatar Sep 18 '23 10:09 nnsgmsone

session buffer的pr已经合入

nnsgmsone avatar Sep 21 '23 10:09 nnsgmsone

先处理mpool的bug

nnsgmsone avatar Sep 26 '23 02:09 nnsgmsone

先处理多session insert报错和daily run的问题

nnsgmsone avatar Oct 07 '23 11:10 nnsgmsone

写软著中。。cache已经实现了一个，不过还没合入mo

nnsgmsone avatar Oct 12 '23 10:10 nnsgmsone

在处理s-1。。。还未处理

nnsgmsone avatar Oct 17 '23 10:10 nnsgmsone

在处理s-1。。。还未处理

nnsgmsone avatar Oct 20 '23 12:10 nnsgmsone

开始编写代码，重新实现新的cache

nnsgmsone avatar Oct 25 '23 10:10 nnsgmsone

明天开始评审第一个pr

nnsgmsone avatar Oct 30 '23 10:10 nnsgmsone

继续修改代码中

nnsgmsone avatar Nov 02 '23 10:11 nnsgmsone

今天在优先处理load的问题。

nnsgmsone avatar Nov 07 '23 10:11 nnsgmsone

定位gap的内存(golang没有统计)中

nnsgmsone avatar Nov 10 '23 11:11 nnsgmsone

继续处理中，具体的问题为https://github.com/matrixorigin/matrixone/issues/12731

nnsgmsone avatar Nov 15 '23 10:11 nnsgmsone

等待测试，然后提交pr

nnsgmsone avatar Nov 20 '23 10:11 nnsgmsone

继续处理mem分支的morpc相关的问题中。

nnsgmsone avatar Nov 23 '23 10:11 nnsgmsone

经过测试eks和129都可以跑，https://github.com/matrixorigin/mo-auto-test/actions/runs/7015615926/job/19085534500。tke的stream closed在等复现，然后fix。

nnsgmsone avatar Nov 28 '23 10:11 nnsgmsone

定位分支tpcc性能下降的问题中。

nnsgmsone avatar Dec 01 '23 10:12 nnsgmsone

今天在修正#13219以及增加更多的metric中。

nnsgmsone avatar Dec 06 '23 10:12 nnsgmsone

no process

nnsgmsone avatar Dec 08 '23 10:12 nnsgmsone

no process

nnsgmsone avatar Dec 13 '23 10:12 nnsgmsone

wait https://github.com/matrixorigin/matrixone/issues/12532

nnsgmsone avatar Dec 20 '23 10:12 nnsgmsone

wait https://github.com/matrixorigin/matrixone/issues/12532

nnsgmsone avatar Dec 25 '23 10:12 nnsgmsone

wait https://github.com/matrixorigin/matrixone/issues/12532

nnsgmsone avatar Dec 28 '23 10:12 nnsgmsone

正在和存储的同事协商https://github.com/matrixorigin/matrixone/issues/12532

nnsgmsone avatar Jan 03 '24 10:01 nnsgmsone