ianb
Automagic unit tests! (support function arguments, and refactor)
Apologies for batching a few changes together, but the gist of the motivation is available in the README:
- Needed a way of accepting function arguments to the magic functions
- Improved the prompt generation by adding argument names for functions, classes
- I noticed that some OpenAI completions only ended in two backticks - fixed that, but not sure how safe it is.
- It's fun being able to run multiple tests at once, all from OpenAI - refactored the code and added an environment variable to allow that
Also making it an environmental variable seems kind of peculiar. I actually think the use case here is worth making a top-level thing with its own prompt
Haven't addressed these two architectural issues yet, but agreed about the environment variable. In the spirit of the original charter for the magic module, having a test function outside magic seems cleanest to me, esp assuming that magic.create_tests(fn)[:5] can't be used to trigger n=5, since the magic fn might return an iterable, and I can't think of another pythonic way of hinting "try multiple times".
test seems to me like it would go outside the magic module, since it differs from both magic and the arrays/dicts.
Not sure on timing for when I could get around to doing that - so I feel like the main question for you is whether we should just remove all the multi-function call stuff and pull it out into a separate PR.