gtfs-validator icon indicating copy to clipboard operation
gtfs-validator copied to clipboard

Analysis improvement: new Github action to trigger comparison between PR acceptance test and past release stats

Open emmambd opened this issue 1 year ago • 0 comments

In cases where we're evaluating changes to the logic of pre-existing rules e.g #1676 and #1675, we want to know the difference between the new PR's acceptance test changes and release analytics from the past, not just the % of feeds triggered by the change relative to all feeds in the Mobility Database. Right now I manually

  • Take the drafted PR's acceptance test results
  • Take the feeds triggered by the notice in the last release analytics
  • Compare the feed IDs of the two to identify
    • Which feeds have dropped/added the notice because of the new drafted logic?
    • Which feeds have dropped/added the notice because of unrelated changes since the last release?
  • Run the individual feeds through https://gtfs-validator.mobilitydata.org/ to see the notice details depending on the particular nuances of that change.

It would be nice in the feed comparison process was automated, so I could see at a glance what % of feeds relative to the number of feeds in the last release that triggered the notice and treat the last release analytics as the source of truth for IDs.

emmambd avatar Mar 06 '24 22:03 emmambd