jasonwhite
Reverse subgraph? (partial builds)
Trying to familiarize myself with Button internals I have attempted to implement a small convenience feature as a practical exercise - being able to specify the resource to build via button build path. It is very useful for CI pipelines, for example to not waste time building release binary and docs if tests fail.
After some investigation I have figured out that all needs to be done is to modify subgraph in button.cli.build.update function to only include vertices that specified resource is reachable from. subgraph.subgraph([ specifiedResourceID ], [ ]) seemed like a good match but experiment has shown it works other way around - provides subgraph of all vertices that can be reached from specified one.
That got me curious if I am not trying to do something unreasonable - was such functionality simply never needed so far or it was omitted intentionally?
Yes, Button differs from build systems like Make in that the build proceeds from the inputs to the outputs. Tup and other modern build systems do it this way too.
There should never be a need to do a partial build. Doing so is bad in terms of correctness and consistency. If you want things to be fast, then incremental builds should solve this problem.
If your release binaries and docs depend on the success or failure of the tests, then that dependency should be specified in the build graph or hoisted out into a separate stage of your CI pipeline.
With that said, it is possible to do this with a bit of code. You'd basically reverse the edges on the fly and change how the graph is traversed in /source/button/graph.d. However, in my opinion it is really an anti-feature that encourages bad practices.
What is the reason it is considered a bad practice? AFAIR any extra resources detected during the bottom-up build can't affect the build graph, only rebuilding criteria. I remember tup doesn't do that either but on the other hand bazel (https://bazel.build) does provide such functionality and I have assumed it uses similar principle.
or hoisted out into a separate stage of your CI pipeline
That was actually the intention - but I can't make it stop until full build succeeds/fails ;) Do you envision creating separate button.json files for such stages?
Well, I consider it bad practice because it means your build graph is partially up-to-date and may hide the fact that a fully up-to-date build graph is failing. For example, suppose you change an interface that libraries A and B consume. If you only change library B to use the new interface and only build library B, then the partial build graph starting from the outputs might succeed whereas the partial graph starting from the inputs would show the failure.
The point of continuous integration is to find problems as fast as possible. Wanting to only build a subset of the outputs indicates to me that the un-built outputs shouldn't be specified in the build description in the first place.
Even though I consider this bad practice, it's probably a good idea to implement it anyway. I know you won't be the only one wanting this. I'm thinking now that the best way to implement this is to take the subgraph two times:
- The subgraph starting from the inputs.
- The subgraph starting from some specified outputs of the first subgraph.
Then, the build can proceed as normal. This should be relatively easy to implement and not require changing the graph traversal algorithm.
I can't make it stop until full build succeeds/fails
With incremental builds, getting to a success or failure state shouldn't take much time. Since incremental builds are designed to be 100% correct (or very close to it), you should never be waiting on a full build.
If you go with the separate stages approach in your CI pipeline, then yes you will need separate button.json files.
If you go with the separate stages approach in your CI pipeline, then yes you will need separate button.json files.
This is actually making quite some sense now that I have experimented with adjusting project layouts to use it. I will still look into implementing partial build support but may not use it myself in the end.
Though my attempt to use such recursive build layout seems to have uncovered strange behavior in button, created separate issue for that https://github.com/jasonwhite/button/issues/17