segmentio
test: ctlstore oom
I have opened this PR for experimental purposes to resolve the OOM issue in warehouses. Once I get approval for my approach, I will add other things such as test cases, and this PR is currently in draft mode. Jira id : https://segment.atlassian.net/browse/CONSENT-139
Purpose :
in tsub library, currently we are using https://github.com/segmentio/ctlstore/blob/146e40069966b5206a427f1c4bbbad7212bd9dc9/ldb_reader.go#L144 function to load all rules based on key array(which is array of primary keys, 1st entry represents scope, 2nd entry represents target_type and 3rd represents target_id), (we are using 1st entry for scope as a key)
and query is
SELECT * FROM tsub_store___rules_materialized_2 WHERE scope = ?
but this is loading ~83k data in cache map for scope=destinations. and in every ttl expiry, we are again loading all scope=destinations based rules.
which is causing OOM in warehouses.
To avoid this issue we want to load only warehouseId specific rules only in cache map for warehouses. which will require targetId like %destinationId% query.
Because this method only provides exact match using = operator. So, introduced a new method for like query.
so, now new function will generate & execute below query
SELECT * FROM tsub_store___rules_materialized_2 WHERE target_id LIKE %destinationId%
Summary mainly memory consumption is in storing and deserializing process of cacheRules(..) function in tsub's ctlstore.go
we are currently loading ~83K rows from db storing them and deserializing each rows in every ttl expiry
this is the main cause of OOM in warehouses, other storage destinations are fine with it
if i reduce this number from 83K to ~2 digits number by not loading all rules based on scope but by loading all rules based on warehouseId.. then no OOM in warehouses as well
