Bulk Loading Part 1: Transaction-Based User Interface
Design principles to make it simple and avoid error-prone:
- Bulk loading is managed based on KeyRange. The bulk loading UI must guarantee that a bulk loading task is running on at most one KeyRange, and a KeyRange can have at most one bulk loading task at a time.
- Users can never overwrite a key range that is running a bulk loading task, unless the user has manually cancelled it.
- Even for a key range which bulk loading task has been completed, users cannot start a new task overlapping the range unless users have explicitly removed the old task on the range.
- When users input a range to remove or cancel bulk loading tasks, the system must cancel all tasks intersecting the input range.
- Each transaction issued by users can only include one of the three types of the operation: new task, check status, cancel task.
User interface:
Say, tr is a readwrite transaction. Each transaction can only include one of the three following types of operation:
- New task:
tr->set("\xff\xff/bulk_loading/task/", bulkLoadStateValue(range, path)); wait(tr->commit());Succeed if no existing bulk loading task (in metadata) intersecting the input task range; Otherwise, throwing an error. - Check status:
wait(tr->getRange("\xff\xff/bulk_loading/status/a", "\xff\xff/bulk_loading/status/b"));Return all tasks's status intersecting the input range. - Cancel task:
tr->clear("\xff\xff/bulk_loading/cancel/a", "\xff\xff/bulk_loading/cancel/b"); wait(tr->commit());Cancel all tasks intersecting the input range.
Those operations to the special key space are mapped to read/write operation on "\xff/bulkLoad/" system key space, which in turn to control the actual bulk loading in FDB internal.
Code-Reviewer Section
The general pull request guidelines can be found here.
Please check each of the following things and check all boxes before accepting a PR.
- [ ] The PR has a description, explaining both the problem and the solution.
- [ ] The description mentions which forms of testing were done and the testing seems reasonable.
- [ ] Every function/class/actor that was touched is reasonably well documented.
For Release-Branches
If this PR is made against a release-branch, please also check the following:
- [ ] This change/bugfix is a cherry-pick from the next younger branch (younger
release-branchormainif this is the youngest branch) - [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)
Result of foundationdb-pr on Linux CentOS 7
- Commit ID: 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
- Duration 0:03:53
- Result: :x: FAILED
- Error:
Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang on Linux CentOS 7
- Commit ID: 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
- Duration 0:03:54
- Result: :x: FAILED
- Error:
Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
- Duration 0:03:56
- Result: :x: FAILED
- Error:
Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
- Commit ID: 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
- Duration 0:04:03
- Result: :x: FAILED
- Error:
Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
- Cluster Test Logs zip file of the test logs (available for 30 days)
Result of foundationdb-pr-macos on macOS Ventura 13.x
- Commit ID: 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
- Duration 0:10:37
- Result: :x: FAILED
- Error:
reference not found for primary source and source version 192b0b19d7031687fb7a3f65c0e86919cccd5b7d - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
- Commit ID: 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
- Duration 0:10:53
- Result: :x: FAILED
- Error:
reference not found for primary source and source version 192b0b19d7031687fb7a3f65c0e86919cccd5b7d - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: 4831a3832bf2f93a6f454a2619c5e1d9ebf9d131
- Duration 0:24:58
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: 42f624be026e8a25f94e4efa969e47506e24ae0d
- Duration 0:24:48
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang on Linux CentOS 7
- Commit ID: 4831a3832bf2f93a6f454a2619c5e1d9ebf9d131
- Duration 0:29:46
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang on Linux CentOS 7
- Commit ID: 42f624be026e8a25f94e4efa969e47506e24ae0d
- Duration 0:31:03
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
- Commit ID: 4831a3832bf2f93a6f454a2619c5e1d9ebf9d131
- Duration 0:36:47
- Result: :x: FAILED
- Error:
Error while executing command: mvn install:install-file --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -Dfile=packages/fdb-java-${FDB_VERSION}-SNAPSHOT.jar -DgroupId=org.foundationdb -DartifactId=fdb-java -Dversion=${FDB_VERSION}-SNAPSHOT -Dpackaging=jar -DgeneratePom=true. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
- Cluster Test Logs zip file of the test logs (available for 30 days)
Result of foundationdb-pr on Linux CentOS 7
- Commit ID: 4831a3832bf2f93a6f454a2619c5e1d9ebf9d131
- Duration 0:39:22
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
- Commit ID: 42f624be026e8a25f94e4efa969e47506e24ae0d
- Duration 0:36:49
- Result: :x: FAILED
- Error:
Error while executing command: mvn install:install-file --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -Dfile=packages/fdb-java-${FDB_VERSION}-SNAPSHOT.jar -DgroupId=org.foundationdb -DartifactId=fdb-java -Dversion=${FDB_VERSION}-SNAPSHOT -Dpackaging=jar -DgeneratePom=true. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
- Cluster Test Logs zip file of the test logs (available for 30 days)
Result of foundationdb-pr on Linux CentOS 7
- Commit ID: 42f624be026e8a25f94e4efa969e47506e24ae0d
- Duration 0:38:16
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: 50c9bae633de4afb5d3869c4162bc39a80ecfd50
- Duration 0:22:45
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
- Commit ID: 50c9bae633de4afb5d3869c4162bc39a80ecfd50
- Duration 0:34:32
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr on Linux CentOS 7
- Commit ID: 50c9bae633de4afb5d3869c4162bc39a80ecfd50
- Duration 0:37:16
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: b9bdb9b8ca155cb57ab7a7b612570fd931b33989
- Duration 0:22:53
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-macos on macOS Ventura 13.x
- Commit ID: 50c9bae633de4afb5d3869c4162bc39a80ecfd50
- Duration 0:46:24
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang on Linux CentOS 7
- Commit ID: b9bdb9b8ca155cb57ab7a7b612570fd931b33989
- Duration 0:29:12
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang on Linux CentOS 7
- Commit ID: 50c9bae633de4afb5d3869c4162bc39a80ecfd50
- Duration 0:52:57
- Result: :x: FAILED
- Error:
Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr on Linux CentOS 7
- Commit ID: b9bdb9b8ca155cb57ab7a7b612570fd931b33989
- Duration 0:37:28
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: 116780bebcddb3e48a1e3577f9474c1ed8c181f7
- Duration 0:21:50
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
- Commit ID: 50c9bae633de4afb5d3869c4162bc39a80ecfd50
- Duration 1:00:48
- Result: :x: FAILED
- Error:
Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
- Cluster Test Logs zip file of the test logs (available for 30 days)
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
- Commit ID: 116780bebcddb3e48a1e3577f9474c1ed8c181f7
- Duration 0:33:51
- Result: :x: FAILED
- Error:
Error while executing command: mvn install:install-file --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -Dfile=packages/fdb-java-${FDB_VERSION}-SNAPSHOT.jar -DgroupId=org.foundationdb -DartifactId=fdb-java -Dversion=${FDB_VERSION}-SNAPSHOT -Dpackaging=jar -DgeneratePom=true. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
- Cluster Test Logs zip file of the test logs (available for 30 days)
Result of foundationdb-pr on Linux CentOS 7
- Commit ID: 116780bebcddb3e48a1e3577f9474c1ed8c181f7
- Duration 0:34:54
- Result: :x: FAILED
- Error:
Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
- Commit ID: 116780bebcddb3e48a1e3577f9474c1ed8c181f7
- Duration 0:35:51
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-clang on Linux CentOS 7
- Commit ID: 116780bebcddb3e48a1e3577f9474c1ed8c181f7
- Duration 0:37:07
- Result: :x: FAILED
- Error:
Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
- Commit ID: b9bdb9b8ca155cb57ab7a7b612570fd931b33989
- Duration 1:01:44
- Result: :x: FAILED
- Error:
Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1 - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
- Cluster Test Logs zip file of the test logs (available for 30 days)
Result of foundationdb-pr-clang-ide on Linux CentOS 7
- Commit ID: ef6d45cbafda644b3b8c92ed71570eb415cb0a1e
- Duration 0:21:32
- Result: :white_check_mark: SUCCEEDED
- Error:
N/A - Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)