Improvement: Running multiple test files at once

[Oscmage]

It would be lovely to run multiple test files at once similar to the github action.

Something like

fga model test --tests ./tests/*.fga.model

The github action got this feature, example:

      - name: Run Test
        uses: openfga/[email protected]
        with:
          test_files_pattern: "./tests/*.fga.yaml"

[7h3-3mp7y-m4n]

is it okay if I pick this?:)

[ewanharris]

@7h3-3mp7y-m4n I think it would be best for us to define this issue a little better before we ask someone to pick it up. But once we've done that it's all yours!

@Oscmage I'm curious what type of file you're passing to the command, is it a file that solely contains tests (as the command states it supports) or are you passing a store file like this.

My concern is that we may have overloaded the --tests option. It states Tests file Name. The file should have the OpenFGA tests in a valid YAML or JSON format but we generally direct folks to pass a store file (1, 2). By simply allowing multiple files to be passed to --tests, we would be supporting running tests against multiple different authorization models at once which I personally don't see the value in (outside of edge cases like sample-stores)

I believe that the value is more separating tests to help reduce file size, have individual "scenarios" per test file, or team ownership per test file, within a single authorization model project.

I'm wondering whether this issue could be broken down into a few parts that would support running multiple test files and help separate some boundaries between test files vs store files and local testing vs remote testing:

• Introduce a --store-file option that will accept a single store file only, it should parse similar to how --tests does currently
• Refactor --tests option to only support being passed multiple test file (i.e. a yaml or json file that doesn't contain anything other than tests)
  • We could then use this as our split between local vs remote testing that is unclear right now (#335), i.e. if you just use --tests we connect to an OpenFGA server to run them, f you pass --store-file we run locally.
• Support a test_files property in the store file similar to the model_file, tuple_file that exist already

[aaguiarz]

@ewanharris even if i agree that receiving multiple store.fga.yaml files with different models is not very useful, I also don't see why supporting it would be an issue. The reason OpenFGA users end create multiple fga.yaml files is usually because each of them tests a specific set of types (or modules), and in that case it makes sense to reuse the same model, the same tuples, and have different tests per file.

[Oscmage]

@ewanharris I would like to pass multiple files that looks like this https://github.com/openfga/cli/blob/main/example/model.fga.yaml . Or if you have a better way to achieve this I am happy to hear those!

What I am trying to solve is that we have multiple types in a model, we'd just like to split the tests for readability reasons and maintenance reasons. Basically all of the different files that I want to run have the same model_file but they might have different tuples setup for the different type that we want to test.

If I have 10 types it is natural to have 10 different test files to easily see the tests for each type. One huge test file makes it harder to read/maintain and reason about.

Happy to elaborate more if this is not clear enough. Again, there might be better ways to solve this problem, very open for that!

[ewanharris]

@Oscmage thanks for that feedback! I had imagine that was your usecase but just wanted to confirm.

I put together what the files could look like based on my comment this gist, instead of storing tests in a store file and running multiple store files you would instead list out your test files similar to how we support a model_file and tuple_file. You could then separate tests whatever fits bets, and if you're using modular models, you could colocate these test files with the various modules.

The main benefit I see of splitting into tests files over multiple store files is that you wouldn't need to copy around the model_file declaration every time you make a new test file (and also update the path if they are in different places). We'd continue to maintain a Store as a single top level entity in OpenFGA also which maybe would help with reasoning about things better.

[ewanharris]

Hey @7h3-3mp7y-m4n, apologies for the delay in circling back for this. If you're still interested in tackling this, we've figured out where this issue should head, I'll try and summarise below.

• Introduce a new test_files field within a store file
  • This would be on the StoreData type internally
  • You can use the existing tuple_file as a reference for this, but we should make sure that if test and test_files are specified they get merged rather than only one being taken
• Each value within test_files should be a path to the file (and we should resolve against the store file, not current directory)
  • If a file doesn't exist, we should error and inform the user of the file that failed to resolve (both the test_files value and resolved path)
• Each file should be a ModelTest
  • It might be worthwhile introducing a similar testfile package to read similar to the tuplefile one that exists

There's an example of what an expected setup will be in this gist. If you have any questions please feel free to ping me (it might be easier to chat on Slack)

[7h3-3mp7y-m4n]

no worries @ewanharris , I'll be happy to work on this, and thank you for summarizing it for me, Yeah I'll get in touch with you on slack :)

[aaguiarz]

We have an example here https://github.com/openfga/sample-stores/pull/53 for where this will be very useful.

We just need to run a set of tests. They do not need to share model/tuples, as that sharing is structured in the fga.yaml files. You can run the tests independently or all together.

I'd like to just have fga model test --tests ./tests/*.fga.yaml

[JarrodJ83]

Any update on this? We too split our test files to make them smaller, more organized, and more readable. This would help us out significantly!

[aaguiarz]

Implemented here https://github.com/openfga/cli/pull/515, we'll release a new version soon