Consider changing RootPathFilter to accept pattern matching

[jlperkins]

We have a request to change RootPathFilter's accepted values from a semicolon-separated list of path prefixes, to file matching patterns. This would help 1ES PT to seamlessly route through the correct files for validation for customers that download pipeline artifacts. This would be a breaking change for us if we replace RootPathFilter. We could consider adding a new option instead of replacing this one, to avoid introducing a breaking change.

[katydecorah]

Thanks for starting this issue, Jules!

I think the tool would need to keep the current behavior of RootPathFilter where it accepts semicolon-separated list of path prefixes since this is the pattern used for downloading partial artifacts drops, and then add the ability to accept file matching (glob patterns), for customers that download partial drops with DownloadPipelineArtifacts, itemPattern property.

I'm not sure if it's too complex to make RootPathFilter to support both types of patterns or to introduce a new property to support file matching patterns.

1ES PT currently has about 500 pipelines that use itemPattern with DownloadPipelineAritfacts. Our goal is to eventually enable SbomValidationFailureEventAction: fail on all release pipelines, and this is a blocker.

The alternative would be for the customer to define both patterns, but this would be nearly impossible to enforce for existing customers and would add significant development toil in translating the two matching patterns.

[katydecorah]

I'm curious if the combination of SbomValidationFailureEventAction: fail and SBOMVALIDATOR_TEMPIGNOREMISSING=true (which implements IgnoreMissing), would be a suitable workaround. From some testing, it seems like the task would not throw an error if there are files in manifest that are missing on disc, meanwhile it would block any other issues.