connelldave
A PyTest-style CLI to run cove-annotated functions
To make botocove useful as a data collection tool you still need to write a Python program around it to format and output the results. I believe it would be helpful to have a generalized implementation of this so that as a user I just have to think about the results I want to return from the botocove function.
PyTest is a good example of how to make the domain code clean and declarative because all the plumbing of detecting tests, running them, and presenting the results is in the PyTest CLI itself. All we as users have to provide are some test functions and fixtures.
This would be a generalization of what I'm trying to achieve with my tool aws-org-inventory.
The command line interface of aws-org-inventory is convenient, but it can query only one API at a time. This makes it really slow to collect info on multiple resources because it requires multiple scans of the organization.
Another approach could be a general CLI interface to botocove. It would run the cove-annotated function in a given file and output a JSON representation of the main result object. This would allow me to write a function that collects all the data in a single session and would allow me to format it appropriately for a posterior JSON-to-CSV convertion.
I've started playing around with this on a branch, with a bit of a combo of both: make botocove when executed directly present a CLI that can collect and run cove_*.py files from a directory/tree, run the funcs and collate per account ID, with a terraform style plan/apply to show the work that'll be done. Bit early to share but feels like a nice value-add!
I have something like this that I've been using repeatedly on my recently analyses.
it's not what I described here, but it's a standard CLI interface that I copy and paste around that exposes the cove options I use and allows me to run a function in multiple orgs in a single command and it outputs all the results as JSON lines. I write the output to a file so that I can then analyze it with jq or load it into a SQLite database using Datasette's sqlite-utils.
When I have time I'll try to extract the generic parts into a reusable module that sits on top of botocove.
But it's Christmas now, so I'll be enjoying some time off for the next couple of weeks.
Merry Christmas and a Happy New Year when it comes!
Hope you had a great Christmas and new year!
I've been thinking about the wider future of this project - realistically, I think it's well placed as a library and can be built into and consumed by other projects. The only change that really impacts is deferring feature conversations to projects built on top of cove as a library, and perhaps loosening the licence. We could then link consuming projects/repos in the readme.
Keen to get your opinion, and potentially save you time if you were looking at shaping a bigger contribution to this codebase?
It makes sense. The core botocove library in this way can be kept small and lean.
For core botocove, which license would you prefer over the current one?
I agree that something like a "CLI for botocove" would be better managed as a separate project that depends on the core botocove library.
I don't have any big changes lined up either way, so this doesn't ruin any plans :-)
In my current work I'm no longer working on organization administration tasks, so I expect to be using botocove less frequently for a while.
I expect it will still be useful for filling in the gaps in AWS standard tools for the accounts I work in
And in any case I'll still be paying attention to what goes on here. You can count on my help for botocove even if I'm no longer a heavy user :-D
Since we've agreed that such a CLI tool is out of scope for the botocove library itself, I'll close this.
For one-shot cove runs, an even more convenient solution may be to read Python code as a command-line argument.
sqlite-utils has a convert command that applies arbitrary transformations to database columns according to the Python code you pass it.
@connelldave I published a Botocove CLI proof of concept. It uses eval to to to return the result of expression passed on the command line.
It's already minimally useful for simple inventory collections. It could be extended with command-line arguments and support for arbitrarily complex input functions.