task and stack tolerance flags do not force a failure and exit.

[zaro0508]

Subject of the issue

task and stack tolerance flags do not force a failure and exit.

Your environment

• version of org-foramtion: 0.9.20-beta.3
• version of node: v16.13.2
• which OS/distro: MAC Monterey 12.3.1

Steps to reproduce

Make sure there's some error in your template. Run print-tasks with --failed-tasks-tolerance 1 --failed-stacks-tolerance 1.

run log

➜ npx org-formation print-tasks ./org-formation/_tasks.yaml --verbose --print-stack --output yaml --max-concurrent-stacks 100 --max-concurrent-tasks 100 --profile admin  --failed-tasks-tolerance 1  --failed-stacks-tolerance 1
DEBG: Task OrganizationUpdate print starting...
INFO: Executing: update-organization org-formation/organization.yaml.
INFO: organization up to date, no work to be done.
INFO: Task OrganizationUpdate print successful.
DEBG: Task Types print starting...
DEBG: Done performing task(s).
INFO: Task IdentityProviders print successful.
INFO: Task Ec2EbsEncryptionDefaults print successful.
DEBG: Task Ec2NoVpcDefaults print starting...
INFO: Task GuardDuty print successful.
DEBG: Task GuardDutyConnectStrides print starting...
ERROR: Task CloudwatchTrailLog print failed. reason: unable to resolve account attribute DnTDevAccount.Tags.CloudwatchCloudTrailLogRetentionPeriod. Tag CloudwatchCloudTrailLogRetentionPeriod not found on account
ERROR:
ERROR: ==========================
ERROR: Stopped performing task(s)
ERROR: Following tasks failed:
ERROR:  - Task CloudwatchTrailLog
ERROR: Following tasks were not executed:
ERROR:  - Task CloudTrail
ERROR:  - Task CloudTrailSumoLogicRole
ERROR: ==========================
ERROR:
ERROR: Task Cloudtrail print failed. reason: Number of failed tasks 1 exceeded tolerance for failed tasks 0.
INFO: Task PreventDisableCloudtrail print successful.
DEBG: Task PreventDisableCloudwatchConfigs print starting...
INFO: Task UnionStationVpc print successful.
DEBG: Task TransitGatewayRoutes print starting...
INFO: Task SecurityHubNotifications print successful.
DEBG: Done performing task(s).
INFO: Task TransitGateway print successful.
DEBG: Task AwsSso print starting...
DEBG: Task SsoAdministrator print starting...
INFO: Task SsoAdministrator print successful.
..
..
DEBG: Task SsoIncludeChopProdDeveloper print starting...
INFO: Task SsoIncludeChopProdDeveloper print successful.
DEBG: Done performing task(s).
INFO: Task AwsSso print successful.
WARN:
WARN: ========================
WARN: Done performing task(s): 1 failed but did not exceed tolerance for failed tasks 1
WARN: Following tasks failed:
WARN:  - Task Cloudtrail
WARN: ========================
WARN:
➜  echo $?
0

Expected behaviour

It should fail and exit on first error that occurs.

Actual behaviour

There are error messages but it does not fail.

[zaro0508]

the failed flags seem to work as expected when setting max concurrent task and stacks to 1 (i.e. --max-concurrent-stacks 1 --max-concurrent-tasks 1)

[OlafConijn]

I would expect the behaviour that is described in this issue: if you run 5 tasks concurrently, you might get up to 5 failures with --failed-tasks-tolerance 1. The reason is that 5 tasks are started (and therefore might fail) before the first task has actually failed.

[zaro0508]

@OlafConijn I would like to make orgf fail fast when executing the print-task or perform-task in parallel, is there a way to do that?