plugin-sdk icon indicating copy to clipboard operation
plugin-sdk copied to clipboard

fix: Properly handle records with multiple rows in batching

Open candiduslynx opened this issue 1 year ago • 1 comments

Fixes https://github.com/cloudquery/cloudquery-issues/issues/1567

candiduslynx avatar Apr 22 '24 17:04 candiduslynx

⏱️ Benchmark results

Comparing with dc31c3aa8639250df277de8b857727b88e503d3b

  • Glob-8 ns/op: 94.5 ⬇️ 2.71% decrease vs. dc31c3aa8639250df277de8b857727b88e503d3b

github-actions[bot] avatar Apr 22 '24 17:04 github-actions[bot]

TODO: Consider making a slicer to handle batching by rows & size. This should produce several records if the new appended record has too much data (e.g. batch size is 5, but we get a record with 100 rows).

This is complicated by the fact that different batchers use different messages, so an overhaul might be required for this.

candiduslynx avatar May 11 '24 18:05 candiduslynx