RBAC Filters Cause Intermittant Crashes

[VandalSausage]

I have a possible reason for this failure after diffing through the src for a few days but I can't quite isolate it so I figured I'd create an issue. I'll add detail as I continue to investigate (might fork to add additional logging).

What version of GraphJin are you using? 0.20.0

Have you tried reproducing the issue with the latest release? No

What is the hardware spec (RAM, OS)? Mac 16gb/ default golang aws lambda

Steps to reproduce the issue (config used to run GraphJin).

Essentially when I add a filter to a role, so using the example of the webshop, add a filter by user id on the product query, then I occasionally get an error that something like "column 'products.user_id' not found"

However this only happens when I have loads of filters (~15) and LOADS of tables across a lot of schemas in my db (around 300 total tables across 20 schemas).

Which leads me to think the filter validation might be being called before the database discovery is fully finished?

It's happening 95% in the lambda we've set up (using GraphJin as a service) but it occassionally happens in the local version I've got running.

I'm actively investigating so will add more detail as I find it.

[dosco]

Can you join the discord feel free to DM me if you don't want to share any queries publicly.

However this only happens when I have loads of filters (~15) and LOADS of tables across a lot of schemas in my db (around 300 total tables across 20 schemas).

Is this with a single query or just in general in the db and config?

Which leads me to think the filter validation might be being called before the database discovery is fully finished?

Filter are compiled at startup time? Are you seeing this error on startup or when executing a query?

NOTE: Queries are compiled on demand and these above pre-compiled filters are inserted into the query at compile time.

[VandalSausage]

Can you join the discord feel free to DM me if you don't want to share any queries publicly.

Yeah for sure

Is this with a single query or just in general in the db and config?

This on a call of 'core.NewGraphjin(config, dbWriter)' in the lambda we're running it in, so it's when the instance recieves the the config struct, finds a bunch of filters and calls 'compileFilter', it then uses 'getColumn' from the internal sdata package to validate the columns used in the filter and that's where this error gets thrown

So to me it looks like the column lookup used in that helper hasn't been fully instantiated by that time the filters are compiled.

Filter are compiled at startup time? Are you seeing this error on startup or when executing a query? NOTE: Queries are compiled on demand and these above pre-compiled filters are inserted into the query at compile time.

Yep to be clear this is an issue with filter compilation at start time, it never gets to the point of being able process a query

[dosco]

Are you still able to recreate this? The initSchema and initDiscover functions are run well before the addRoles in called in initCompilers so this is probably something else and not that the db discovery and init has not completed. I'd love to help fix this.

[VandalSausage]

Yep, we've made no progress with this. It's intermittent but happens around 90% of the time.

It's only present when there is a filter on a table for a role in the config bundle.

The frequency of occurrence seems to be directly related to the number of columns in the db.

If you have any suggestions for things we could investigate to help diagnose the issue that would be great :)

[dosco]

I'll create a 1000 column table and run some tests.

[VandalSausage]

Our set up is ~40 schemas, each with ~10 tables, each of those with 5 - 10 columns

[dosco]

I know its been a while but I had some time to look into this and created a test to generate 100 schema with like 10 tables each and try to recreate this issue. Since I don't personally use Schemas I've never dug into this feature as much. How do you write the graphql to ask for a table within a specific schema? We don't have support to specify the schema to use maybe we need this.

[dosco]

@VandalSausage I have added a test for your issue and some fixes including a new @schema directive to specify what db schema the selector referrs to. https://github.com/dosco/graphjin/blob/dfe77a966d8ebbdac739c28b2d5308a191038328/core/postgres_test.go

[dosco]

@VandalSausage If you are still experiencing this issue please reopen. I have a fix and text in place not sure if it works for you.