trailofbits
Create a tabular output for run-all-attacks
Is your feature request related to a problem? Please describe. When running multiple attacks, it should be easy to compare the effectiveness of each one.
Describe the solution you'd like. Returning a table of each attack and respective metrics is a start. This could eventually be turned into graphs and other visuals. A pandas data frame is a potential format.
Detail any additional context. TensorFlow Privacy is an excellent example.
I'm thinking about working on this. Still introducing myself to the codebase. When you say run-all-attacks, is that run_all_extraction()?
It seems like I'd build this table out of the various ModelExtractionAttack.label_agreement attributes after for s in synths. Does that sound right?
Thank you so much for your interest in PrivacyRaven!
That sounds correct! Right now, unfortunately, this issue is blocked on #79 (the new library updates means that the HopSkipJump-based extraction attack no longer works). However, if you're still interested in contributing to PrivacyRaven, here's the project plan and more good first issues.
Feel free to contact me through the Empire Hacking Slack or at [email protected] if you'd like a tour of the codebase or you want to talk about PrivacyRaven more.