fides
fides copied to clipboard
ENG-762 add manual task to privacy req
Closes ENG-762
Description Of Changes
Updates aspects of Privacy Requests and DSR access packages to work with manual tasks.
Execution Flow with Data Access
-
Pre-execution tasks:
- Connect to root (run early)
- Connect to first data nodes (provide data to downstream tasks)
- Run in parallel with each other
-
Regular data processing tasks:
- Run according to their existing dependencies
- Can access data from pre-execution tasks
- Connect to post-execution tasks (provide data to post-execution tasks)
-
Post-execution tasks:
- Connect to all data processing nodes (receive data from upstream tasks)
- Connect to terminator (run late)
- Run in parallel with each other
Data Access Matrix
| Task Type | Can Access Pre-execution | Can Access Regular Tasks | Can Access Post-execution |
|---|---|---|---|
| Pre-execution | ❌ (run in parallel) | ❌ (run before) | ❌ (run before) |
| Regular tasks | ✅ (via edges) | ✅ (via existing dependencies) | ❌ (run before) |
| Post-execution | ✅ (via edges) | ✅ (via edges) | ❌ (run in parallel) |
This ensures that:
- Pre-execution tasks provide data to downstream tasks
- Regular tasks can access pre-execution data and provide data to post-execution tasks
- Post-execution tasks can access data from both pre-execution and regular execution tasks
- Execution order is maintained: pre-execution → data processing → post-execution
- Parallel execution is maintained within each timing group
Code Changes
- Manual tasks each get their own node on the dag
- There are pre execution nodes which must be comepleted before other tasks can run
- There are post execution nodes which must be completed after other nodes have run.
Steps to Confirm
- list any manual steps for reviewers to confirm the changes
Pre-Merge Checklist
- [ ] Issue requirements met
- [ ] All CI pipelines succeeded
- [ ]
CHANGELOG.mdupdated- [ ] Add a https://github.com/ethyca/fides/labels/db-migration label to the entry if your change includes a DB migration
- [ ] Add a https://github.com/ethyca/fides/labels/high-risk label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
- [ ] Updates unreleased work already in Changelog, no new entry necessary
- Followup issues:
- [ ] Followup issues created
- [ ] No followup issues
- Database migrations:
- [ ] Ensure that your downrev is up to date with the latest revision on
main - [ ] Ensure that your
downgrade()migration is correct and works- [ ] If a downgrade migration is not possible for this change, please call this out in the PR description!
- [ ] No migrations
- [ ] Ensure that your downrev is up to date with the latest revision on
- Documentation:
- [ ] Documentation complete, PR opened in fidesdocs
- [ ] Documentation issue created in fidesdocs
- [ ] If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
- [ ] No documentation updates required
The latest updates on your projects. Learn more about Vercel for Git ↗︎
2 Skipped Deployments
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| fides-plus-nightly | ⬜️ Ignored (Inspect) | Visit Preview | Jun 20, 2025 10:34pm | |
| fides-privacy-center | ⬜️ Ignored (Inspect) | Jun 20, 2025 10:34pm |
Hi @JadeCara, I'm closing this ticket, I covered it in https://github.com/ethyca/fides/pull/6261. I went with the "parallel" approach to remove the need to specify pre, parallel, or post. Just one less thing for the user to think about. The manual task nodes will show up as "awaiting input" and are essentially paused until the field data is completed or skipped.