incubator-devlake
incubator-devlake copied to clipboard
[Refactor][core]Data inflation when using postgres
What and why to refactor
As a software engineer, I am using devlake to collect developer data since long time ago. Recently we find that the _devlake_subtasks table occurs a data inflation. It used 750MB after a week from we upgrading devlake v1.0, but there were only 1000 records in the table.
Describe the solution you'd like
Solutions from @klesh
- Reduce the update rate to _devlake_subtasks table when collect data.
- Store progress info in memory firstly and write it to db in a fixed rate.
Related issues
No
Additional context
- The issue was introduced in v1.0.
- We are using postgres.