sybil icon indicating copy to clipboard operation
sybil copied to clipboard

Add block skipping for string filters

Open okayzed opened this issue 5 years ago • 2 comments

If filtering to a particular string and the block doesn't contain that string, we can skip aggregating that block. This might help certain use cases for redbull.

Basically, we would prioritize unpacking that string column first and then check filter against the string table.

This may or may not work well.

okayzed avatar Mar 06 '20 13:03 okayzed

evan suggests we can do this using bloom filters and hierarchical bloom filters. this will work for equality but not regex, as far as i can tell

okayzed avatar Mar 07 '20 15:03 okayzed

An initial implementation now exists for simple equality on strings on a per block basis, but it is not using bloom filters

okayzed avatar Apr 11 '20 18:04 okayzed