DataBiosphere
Attach custom tag to AWS EC2 when running toil-cwl-runner
Is there a way to set custom tag for the AWS EC2 instances run with toil-cwl-runner? I see there is the TOIL_OWNER_TAG, but this is only for launching clusters. The reason being that we'd would for example set up the lead node, but from there run workflows sequentially. But if we can set the tags upon each run, we can easily monitor and estimate costs, etc via tag filtering.
┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1223
@Hexotical I don't think this is the same idea as @DailyDreaming was talking about in https://github.com/DataBiosphere/toil/pull/4220#pullrequestreview-1115815198, with different tags for e.g. EC2 vs. S3. This issue is about letting you start one Toil cluster or leader, and then running multiple workflows on it, with each workflow running on workers with particular workflow-specific tags.
@boyangzhao I think that #4220 will allow for this, if you are using Toil's in-workflow autoscaling, where each workflow runs a provisioner. You could set the new tagging environment variable for each workflow, and then when that workflow's provisioner makes nodes, they would be tagged as specified in that variable.
This isn't going to be possible with the newer Kubernetes-integrated autoscaling that we like somewhat better, though. That relies on setting up the cluster in advance with the kinds of nodes it can have, and letting the cluster scale itself up and down using Kubernetes's Cluster Autoscaler. That setup lets us run multiple simultaneous workflows, and use autoscaling code that we aren't responsible for ourselves, but it means that the nodes aren't connected to the individual workflows in a way that would let them be tagged per-workflow.
Ah nice! This means I can launch-cluster without or without the tag; but later when I run with toil-cwl-runner, it'll take the tagging environment variable at the time to tag all the subsequent worker nodes of that run? I see that launch-cluster has the --tag argument as well. But toil-cwl-runner will not have it's own --tag argument, so this needs to be defined by the env var?