Introduction AuditStorage is a functionality that serves to audit the system's data storage by checking for data consistency. It is triggered when a client requests an audit or when a consistency check is required at the end of simulation. When a client issues an auditStorage request, the request is first processed by CC. CC forwards the request to DD, which is responsible for processing audit requests. DD checks for ongoing audits before processing a new audit request. If there is an ongoing audit with the same audit type and range as the new request, DD obtains the audit ID for that audit. If there is an ongoing but irrelevant audit, DD returns an error message indicating that the system is busy, as currently only one ongoing audit is allowed at a time. If there is no ongoing audit, DD creates a new audit and persists its state. AuditStorage requests are asynchronous. DD immediately replies with the existing audit ID to CC. If DD is unable to get and persist the result, it captures the failure and automatically retries the audit until one of three outcomes is achieved: (1) the maximum number of retry attempts is exceeded, resulting in a "Failed" result; (2) the audit is completed without any errors, resulting in a "Complete" result; or (3) the audit is completed with errors detected, resulting in an "Error" result. In some cases, CC might not know whether the request has been delivered to DD, for example, when DD restarts after CC sends an audit request. In such cases, CC replies with "request_maybe_delivered" to the client. The client can then issue a new audit request if necessary.

Following designs are obeyed when developing the audit storage: (1) A audit request generates an AuditStorage; (2) An AuditStorage can automatically retry for failures; (3) Any component of AuditStorage must not block or kill SS and DD; (4) Audit storage must be retriable --- being able to make progress by retrying. A large audit is partitioned into tasks and assigned to SSes. Each SS runs assigned tasks until completing all assigned tasks or failed. Upon completing each task, SS persists the progress. If a task is failed, SS notifies DD, and DD loads the progress made by the SS and resend the remaining tasks to the SS.

Current limitations: (1) TSS servers are not covered; (2) If a bad assignment consistently updated to metadata, this bad assignment is not detected. For example, DD assigns an empty SS or a removed SS to KeyServer and ServerKey. This bad assignment cannot be detected by current implementation of AuditStorage.

AduitStorageTest 100k: 20230429-063732-zhewang-b06297784516243e compressed=True data_size=32945656 duration=3028909 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:45:18 sanity=False started=100000 stopped=20230429-072250 submitted=20230429-063732 timeout=5400 username=zhewang

100k correctness: 20230429-063413-zhewang-422a958e38f9fa0b compressed=True data_size=32915465 duration=5142264 ended=100000 fail=2 fail_fast=10 max_runs=100000 pass=99998 priority=100 remaining=0 runtime=1:19:41 sanity=False started=100000 stopped=20230429-075354 submitted=20230429-063413 timeout=5400 username=zhewang

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

[ ] The PR has a description, explaining both the problem and the solution.
[ ] The description mentions which forms of testing were done and the testing seems reasonable.
[ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

[ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
[ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

Mar 08 '23 22:03 kakaiu