BCDA-4663/4696 Feature: Python script to flatten NDJSON -> CSV

[jphechter]

Work for BCDA-4663 and BCDA-4696

Proposed Changes

• Create a directory for scripts relevant to the application, but not part of primary tooling
• Add a script to convert NDJSON structured data to a single flat CSV
• Update .gitignore
• add README to accompany this script

Change Details

The script works by flattening nested JSON data, and then builds a CSV.

Security Implications

• [ ] new software dependencies
• [ ] security controls or supporting software altered
• [ ] new data stored or transmitted
• [ ] security checklist is completed for this change
• [ ] requires more information or team discussion to evaluate security implications
• [X] no PHI/PII is affected by this change

Acceptance Validation

There are no tests for this work currently. This is a standalone script that is meant to be more of a stopgap solution than a fully fledged feature.

• Review logic of the script
• Recommend following the included README to build Python venv, install package dependencies, and test script
• Review output for all 3 resource types
  • I used data from the sandbox environment to build this. Recommend pulling the same data for testing purposes. I can also save you the time and upload that somewhere.
• Need to manually generate ALR data by running test with output flag: go test -v github.com/CMSgov/bcda-app/bcda/models/fhir/alr -run TestGenerateAlr -output -version 1

Feedback Requested

Open to all forms of feedback and flying accusations. If there's a better way to approach this, I'd love to hear it.

[codecov-commenter]

Codecov Report

Merging #727 (4c55fce) into master (d669683) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #727   +/-   ##
=======================================
  Coverage   64.93%   64.93%           
=======================================
  Files          64       64           
  Lines        5539     5539           
=======================================
  Hits         3597     3597           
  Misses       1495     1495           
  Partials      447      447           

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d669683...4c55fce. Read the comment docs.

[carlpartridge]

Closing due to inactivity.