plugin-sdk
plugin-sdk copied to clipboard
fix: Properly handle records with multiple rows in batching
Fixes https://github.com/cloudquery/cloudquery-issues/issues/1567
⏱️ Benchmark results
Comparing with dc31c3aa8639250df277de8b857727b88e503d3b
- Glob-8 ns/op: 94.5 ⬇️ 2.71% decrease vs. dc31c3aa8639250df277de8b857727b88e503d3b
TODO: Consider making a slicer to handle batching by rows & size. This should produce several records if the new appended record has too much data (e.g. batch size is 5, but we get a record with 100 rows).
This is complicated by the fact that different batchers use different messages, so an overhaul might be required for this.