fides icon indicating copy to clipboard operation
fides copied to clipboard

ENG-762 add manual task to privacy req

Open JadeCara opened this issue 7 months ago • 1 comments

Closes ENG-762

Description Of Changes

Updates aspects of Privacy Requests and DSR access packages to work with manual tasks.

Execution Flow with Data Access

  1. Pre-execution tasks:

    • Connect to root (run early)
    • Connect to first data nodes (provide data to downstream tasks)
    • Run in parallel with each other
  2. Regular data processing tasks:

    • Run according to their existing dependencies
    • Can access data from pre-execution tasks
    • Connect to post-execution tasks (provide data to post-execution tasks)
  3. Post-execution tasks:

    • Connect to all data processing nodes (receive data from upstream tasks)
    • Connect to terminator (run late)
    • Run in parallel with each other

Data Access Matrix

Task Type Can Access Pre-execution Can Access Regular Tasks Can Access Post-execution
Pre-execution ❌ (run in parallel) ❌ (run before) ❌ (run before)
Regular tasks ✅ (via edges) ✅ (via existing dependencies) ❌ (run before)
Post-execution ✅ (via edges) ✅ (via edges) ❌ (run in parallel)

This ensures that:

  • Pre-execution tasks provide data to downstream tasks
  • Regular tasks can access pre-execution data and provide data to post-execution tasks
  • Post-execution tasks can access data from both pre-execution and regular execution tasks
  • Execution order is maintained: pre-execution → data processing → post-execution
  • Parallel execution is maintained within each timing group

Code Changes

  • Manual tasks each get their own node on the dag
  • There are pre execution nodes which must be comepleted before other tasks can run
  • There are post execution nodes which must be completed after other nodes have run.

Steps to Confirm

  1. list any manual steps for reviewers to confirm the changes

Pre-Merge Checklist

  • [ ] Issue requirements met
  • [ ] All CI pipelines succeeded
  • [ ] CHANGELOG.md updated
    • [ ] Add a https://github.com/ethyca/fides/labels/db-migration label to the entry if your change includes a DB migration
    • [ ] Add a https://github.com/ethyca/fides/labels/high-risk label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
    • [ ] Updates unreleased work already in Changelog, no new entry necessary
  • Followup issues:
    • [ ] Followup issues created
    • [ ] No followup issues
  • Database migrations:
    • [ ] Ensure that your downrev is up to date with the latest revision on main
    • [ ] Ensure that your downgrade() migration is correct and works
      • [ ] If a downgrade migration is not possible for this change, please call this out in the PR description!
    • [ ] No migrations
  • Documentation:
    • [ ] Documentation complete, PR opened in fidesdocs
    • [ ] Documentation issue created in fidesdocs
    • [ ] If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
    • [ ] No documentation updates required

JadeCara avatar Jun 17 '25 19:06 JadeCara

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Skipped Deployments
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Jun 20, 2025 10:34pm
fides-privacy-center ⬜️ Ignored (Inspect) Jun 20, 2025 10:34pm

vercel[bot] avatar Jun 17 '25 19:06 vercel[bot]

Hi @JadeCara, I'm closing this ticket, I covered it in https://github.com/ethyca/fides/pull/6261. I went with the "parallel" approach to remove the need to specify pre, parallel, or post. Just one less thing for the user to think about. The manual task nodes will show up as "awaiting input" and are essentially paused until the field data is completed or skipped.

galvana avatar Jul 02 '25 19:07 galvana