aws
feat: Support and use Toil 5.8's WES task output
Issue #, if available:
Description of Changes
As of https://github.com/DataBiosphere/toil/pull/4166, due to be released in Toil 5.8, Toil will now report tasks in its WES run log output.
Unfortunately, AGC will crash if it sees a task in WES output where the task name does not end in | and then an AWS Batch job ID. By default Toil does not apply this transformation to task names, even when running against Amazon Batch, because it really only makes sense to do for AGC. So Toil's server now has a --wes_dialect option that can be set to agc to force it to adjust task names to meet this constraint.
This PR updates the Toil version that the Toil engine container is set to be built against, and adjusts the container's script to launch the Toil server with --wes_dialect agc to constrain it to produce only task names that meet AGC's particular requirements.
Description of how you validated changes
I built the Toil server container as specified and ran the hello demo CWL workflow using it, and I was able to see the task information from Toil in agc logs workflow:
[anovak@swords demo-cwl-project]% agc logs workflow hello
2022-08-11T11:40:44-04:00 𝒊 Showing the logs for 'hello'
Assume Role MFA token code: 444071
2022-08-11T11:40:54-04:00 𝒊 Showing logs for the latest run of the workflow. Run id: 'run-06839418265146ccabb3b56d0c0f39a1'
RunId: run-06839418265146ccabb3b56d0c0f39a1
State: COMPLETE
Tasks:
Name JobId StartTime StopTimeExitCode
hello.cwl 3b0f03f4-1106-496f-9f9c-3b0997e2dcd2 <nil> <nil> 0
Run Standard Output:
{
"output": {
"location": "s3://agc-318423852362-us-west-2/project/CWLDemo/userid/anovak4rBHiA/context/myContext/toil-execution/output/run-06839418265146ccabb3b56d0c0f39a1/output.txt",
"basename": "output.txt",
"nameroot": "output",
"nameext": ".txt",
"class": "File",
"checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
"size": 13
}
}
Run Standard Error:
[2022-08-11T15:34:25+0000] [MainThread] [I] [cwltool] Resolved '/opt/work/workflows/run-06839418265146ccabb3b56d0c0f39a1/execution/hello.cwl' to 'file:///opt/work/workflows/run-06839418265146ccabb3b56d0c0f39a1/execution/hello.cwl'
[2022-08-11T15:34:26+0000] [MainThread] [I] [toil] Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
[2022-08-11T15:34:26+0000] [MainThread] [I] [toil] Using default docker name of toil as TOIL_DOCKER_NAME is not set.
[2022-08-11T15:34:26+0000] [MainThread] [I] [toil] Using default docker appliance of quay.io/ucsc_cgl/toil:5.8.0a1-bf2c046b5b1d38bdfc3043a4ea05f72db279ab64-py3.7 as TOIL_APPLIANCE_SELF is not set.
[2022-08-11T15:34:26+0000] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 new
[2022-08-11T15:34:26+0000] [MainThread] [I] [toil.job] Processing job 'CWLJob' hello.cwl 19736ad2-87df-40cc-93eb-5c639a8e8e94 v0
[2022-08-11T15:34:27+0000] [MainThread] [I] [toil] Running Toil version 5.8.0a1-bf2c046b5b1d38bdfc3043a4ea05f72db279ab64 on host ip-10-0-159-188.us-west-2.compute.internal.
[2022-08-11T15:34:27+0000] [MainThread] [I] [toil.leader] Issued job 'CWLJob' hello.cwl 19736ad2-87df-40cc-93eb-5c639a8e8e94 v1 with job batch system ID: 0 and cores: 1, disk: 3.0 Gi, and memory: 2.0 Gi
[2022-08-11T15:34:29+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
[2022-08-11T15:37:56+0000] [MainThread] [I] [toil.leader] Finished toil run successfully.
[2022-08-11T15:37:57+0000] [MainThread] [I] [toil.common] Successfully deleted the job store: <toil.jobStores.aws.jobStore.AWSJobStore object at 0x7ff311ba6ed0>
Toil doesn't yet report task start and stop times, but when it gains that feature they should become visible automatically.
Also included is documentation of the steps needed to build the Toil engine container on an ARM host, because the existing instructions assumed that the user was working from an AMD64 machine.
This can't really be unit tested; it needs a whole running Toil-based context to be seen. Unit tests already exist that mock the WES responses, AFAIK.
Checklist
- [X] If this change would make any existing documentation invalid, I have included those updates within this PR
- N/A I have added unit tests that prove my fix is effective or that my feature works
- [X] I have linted my code before raising the PR
- [X] Title of this Pull Request follows Conventional Commits standard: https://www.conventionalcommits.org/en/v1.0.0/
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
Is there something weird I did in this PR that makes golangci-lint get killed in the CI pipeline? Or is the CI pipeline broken?