AugurProject
Create test suite for template generator.
Note: If this is already done and I am just looking in the wrong place feel free to comment and close.
The template generator is some pretty complicated code and it includes the generation of template validators. This code also is related to end-user security/confidence in that users will likely trust templated markets implicitly and so anyone who can create an invalid template market can scam users easily and greatly reduce confidence in the platform.
The most important thing is that there are tests for the template validators that run against the production templates. This will make it easy for anyone to try to come up with markets that violate the template rules and test to make sure the template validator won't let it through. Without tests, engineers, auditors, and penetration testers will need to manually create markets against a blockchain and then check to see if the UI validates them.
There is a flash script validate-template that is used to manually validate market creation template object. There are examples in flash README.md how to use it.
The "production" templates are stable now. Giving "unit tests" that run the flash script to validate various production templates with variations of inputs might be helpful, but I think the same thing can be accomplished just adding the examples to documentation. This would give users the ability to copy and paste to run the validate-template themselves.
The goal with a test suite is so we can easily test edge/failure cases and prove that they don't validate. For example, a parameterized test might take a market title as input for an invalid market and then expect that isTemplateMarket returns false. This would both allow us to quickly iterate on the validation checking without fear of introducing scam market regressions, while also quickly testing a wide variety of market descriptions/outcomes to test the edge cases.
See #5223 for some very simple examples of edge cases that should be tested.
I agree with a test that runs market created template variations against validate-template flash script. I think that would be useful.