JWock82
Run examples as part of test suite
The examples can be run during tests to surface errors. This does not include making sure the results are correct. This adds another layer of checks to catching any breaking changes to the public API.
A downside is that it lengthens the time it takes to run the test suite. They take approximately 2 minutes and 12 seconds on my machine. This could be addressed by excluding the tests by default, and including them as part of CI.
Let me know and I will open a PR.
I love this idea. I am in the camp that even the python doc strings should be working code and included in the test suite. This helps to prevent stale examples in guides. Pytest has a plug-in that does this called pytest-codeblocks that will pick up markdown files automatically with Python script blocks (https://github.com/nschloe/pytest-codeblocks) or my testing library rustest has this built in (https://github.com/Apex-Engineers-Inc/rustest). The added benefit to rustest is it will speed up the test suite as well.
@JBloss1517 I think running code in the documentation as part of the test suite is fantastic idea! I gave pytest-codeblocks a try but it didn't discover any code in the documentation, I believe we'll need to find an option that support reStructuredText.
rustest looks very cool! I gave it a try, but unfortunately ran into an error. If you open a separate issue, I'd be happy to share the traceback there.