Update "Live compilation result" through Dataform API

[mbgrandi]

I'm wondering if there is any way to update the live compilation result displayed in the release configuration details using the Dataform API? I can easily create compilation results calling:

https://dataform.googleapis.com/v1/projects/*/locations/*/repositories/*/compilationResults

However, we would need a way to update the live compilation result automatically when creating a compilation result. The most frequent schedule we have for automatic updates of the compilation result is hourly, but I wonder if we could update it via a call to the API in a GitHub action for example.

We have several Dataform repos linked to GitHub, and after merging changes to main branch we would like to update the live compilation result automatically. Otherwise, we would need to wait for the next hour for the scheduler "pick up" our changes and generate a fresh compilation result. If the users don't wait for that they could end up running the workflow with outdated code.

According to the docs: "The latest compilation result created for the release configuration is the live compilation result", but that is not true for API calls.

https://cloud.google.com/dataform/docs/release-configurations#release-configuration-process

Is there any other way to do that through the API or some other automation? We know we could hit the "New compilation" button manually, but that would be hard to track and assure it happens every time.

Thanks!

[mbgrandi]

After some testing, I've found the correct way to update the live compilation result using the API. First we create the compilation results using:

https://cloud.google.com/dataform/reference/rest/v1/projects.locations.repositories.compilationResults/create

curl --header "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
     --header 'Content-Type: application/json' \
     --data    '{"releaseConfig": "projects/<your_project_name>/locations/<project_location>/repositories/<your_repository_name>/releaseConfigs/<your_release_config_name>"}' \
     -X POST \
     https://dataform.googleapis.com/v1/projects/<your_project_name>/locations/<project_location>/repositories/<your_repository_name>/compilationResults

After that we need to update the release configuration with the created compilation result:

https://cloud.google.com/dataform/reference/rest/v1/projects.locations.repositories.releaseConfigs/patch

curl --header "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
     --header 'Content-Type: application/json' \
     --data    '{"releaseCompilationResult": "projects/<your_project_name>/locations/<project_location>/repositories/<your_repository_name>/compilationResults/<compilation_result_id>", "gitCommitish": "main"}' \
     -X PATCH \
     https://dataform.googleapis.com/v1/projects/<your_project_name>/locations/<project_location>/repositories/<your_repository_name>/releaseConfigs/<your_release_config_name>

[fernhtls]

hi @mbgrandi, I've bumped into the same problem on our dataform repo when trying to push some automation on releases tied to a github release/tag, but had got to the point of the compilationResults only before. One question though, the api call to compilationResults is async I think, so the call does not wait for the compilation to finish, and then probably in between the calls above I imagine you're doing a loop to get the status of it, and if successful then triggering the update to the live result. Am I correct on my assumption? If not could you tell me what's your flow then. Thanks!

[mbgrandi]

Hey @fernhtls, when you POST the compilationResults you get a response back with your compilation result id which you need for the next call to update the release configuration. So I believe this is using a synchronous call, otherwise you won't get the result id in the response.

We do these calls in a Makefile, using simple curl:

1. Call compilationResults;
2. Get the response and check compilation result id;
3. If we have the id we call the releaseConfigs to update it;
4. Check the response from the second call, you should get the release config instance;
5. If any error happens we throw the error messages;

We can use this locally or call the make target in a GitHub action. Hope that helps! Let me know if you need any questions.

[fernhtls]

Thanks @mbgrandi ! I'll do a few test to check if the compilationResults is async or sync.

[mbgrandi]

Thanks @mbgrandi ! I'll do a few test to check if the compilationResults is async or sync.

Cool, let me know your findings. We hope they improve this in the future, adding this to the CLI would be great.

[fernhtls]

Hi @mbgrandi , just did a new test, and indeed you get the full path of the compilation including the compilation ID, and that can be used to run the API to change the live compilation instantly. In our case the compilation itself is quick, as I've tried on a sample dataform repo. I'll be pushing it to some official repos later on this month, and when I get the results will share again here.

[mbgrandi]

Hi @mbgrandi , just did a new test, and indeed you get the full path of the compilation including the compilation ID, and that can be used to run the API to change the live compilation instantly. In our case the compilation itself is quick, as I've tried on a sample dataform repo. I'll be pushing it to some official repos later on this month, and when I get the results will share again here.

Great that worked well on your tests!