incubator-devlake icon indicating copy to clipboard operation
incubator-devlake copied to clipboard

[Refactor][core]Data inflation when using postgres

Open narrowizard opened this issue 1 year ago • 0 comments

What and why to refactor

As a software engineer, I am using devlake to collect developer data since long time ago. Recently we find that the _devlake_subtasks table occurs a data inflation. It used 750MB after a week from we upgrading devlake v1.0, but there were only 1000 records in the table.

Describe the solution you'd like

Solutions from @klesh

  1. Reduce the update rate to _devlake_subtasks table when collect data.
  2. Store progress info in memory firstly and write it to db in a fixed rate.

Related issues

No

Additional context

  • The issue was introduced in v1.0.
  • We are using postgres.

narrowizard avatar Oct 11 '24 08:10 narrowizard