In the PR https://github.com/apple/foundationdb/pull/11678, we add a validation of an invariant after buildTeam: if latest buildTeam succeeds, then each server must have at least targetTeamNumPerServer serverTeams.

What if the buildTeam consistently failed? The simply idea is that keeping tracking the time span that buildTeam keeps failing. If the time span exceeds a threshold, then trace error.

What makes the question interesting is that: the machine, server, and zone layout may not satisfy the buildTeam. As a result, the buildTeam can be consistently failed. For example, if the machine count per zone is small, buildMachineTeam is likely to be failed.

In this PR, we add a logic to decide if the current machine, server, and zone is satisfiable to the targetTeamNumPerServer. We conduct the detection of buildTeam consistent failing only when the layout is satisfiable.

Conclusion The PR passed 100K tests in release-7.3, indicating if the layout is good, each server eventually has target amount of ServerTeams.

100K correctness: 20241105-052134-zhewang-2503468195659beb compressed=True data_size=35141615 fail_fast=10 max_runs=100000 priority=100 sanity=False submitted=20241105-052134 timeout=5400 username=zhewang

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

[ ] The PR has a description, explaining both the problem and the solution.
[ ] The description mentions which forms of testing were done and the testing seems reasonable.
[ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

[ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
[ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

Sep 26 '24 17:09 kakaiu

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: b9eebcfd6a52bb83abf99072471ea04dad76b1b4
Duration 0:45:53
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Sep 26 '24 18:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: b9eebcfd6a52bb83abf99072471ea04dad76b1b4
Duration 0:55:53
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

Sep 26 '24 18:09 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: b9eebcfd6a52bb83abf99072471ea04dad76b1b4
Duration 0:59:00
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Sep 26 '24 18:09 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

Commit ID: b9eebcfd6a52bb83abf99072471ea04dad76b1b4
Duration 1:13:16
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Sep 26 '24 19:09 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: baebb1fd754da8da6eced6018905b78b250a0519
Duration 0:37:48
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Sep 27 '24 05:09 foundationdb-ci