bugbug icon indicating copy to clipboard operation
bugbug copied to clipboard
verified publisher icon mozilla

Start checking for feature importances evolution. Fix #824

Open Lothiraldan opened this issue 6 years ago • 3 comments

Lothiraldan avatar Sep 26 '19 19:09 Lothiraldan

Codecov Report

Merging #982 into master will increase coverage by 1.63%. The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #982      +/-   ##
==========================================
+ Coverage    49.4%   51.03%   +1.63%     
==========================================
  Files          90       85       -5     
  Lines        6285     5821     -464     
==========================================
- Hits         3105     2971     -134     
+ Misses       3180     2850     -330
Impacted Files Coverage Δ
scripts/analyze_training_metrics.py 0% <0%> (ø) :arrow_up:
bugbug/test_scheduling.py 0% <0%> (-80%) :arrow_down:
bugbug/models/defect_enhancement_task.py 27.27% <0%> (-31.82%) :arrow_down:
bugbug/models/defect.py 65% <0%> (-17.5%) :arrow_down:
bugbug/models/regressor.py 32.55% <0%> (-4.95%) :arrow_down:
bugbug/labels.py 92% <0%> (-3.24%) :arrow_down:
bugbug/utils.py 67.11% <0%> (-3.09%) :arrow_down:
bugbug/model.py 69.2% <0%> (-1.81%) :arrow_down:
scripts/trainer.py 74.24% <0%> (-1.15%) :arrow_down:
bugbug/bug_snapshot.py 61.17% <0%> (-0.4%) :arrow_down:
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cf2dc87...8b05cea. Read the comment docs.

codecov-io avatar Sep 26 '19 19:09 codecov-io

Here is the analysis as of today:

WARNING:__main__:Feature for model 'regression' changed between 2019-09-06 15:33:54+00:00 and 2019-09-07 00:06:33+00:00
WARNING:__main__:Feature only present at 2019-09-06 15:33:54+00:00:
{"Comments contain 'str'"}
WARNING:__main__:Feature only present at 2019-09-07 00:06:33+00:00:
set()
WARNING:__main__:Present in both:
{"Comments contain 'nightly'",
 "Comments contain 'regression'",
 "First comment contains 'we'",
 'intermittent-failure in keywords',
 "status has ever been set to 'affected' and 'unaffected'=False"}
WARNING:__main__:Feature for model 'regression' changed between 2019-09-12 08:15:25+00:00 and 2019-09-12 11:26:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 08:15:25+00:00:
set()
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
{"Comments contain 'pushlog'"}
WARNING:__main__:Present in both:
{"Comments contain 'nightly'",
 "Comments contain 'regression'",
 "First comment contains 'we'",
 'intermittent-failure in keywords',
 "status has ever been set to 'affected' and 'unaffected'=False"}
WARNING:__main__:Feature for model 'regression' changed between 2019-09-12 11:26:14+00:00 and 2019-09-16 00:01:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
{"Comments contain 'pushlog'"}
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
{"First comment contains '__file_reference__'", "Comments contain 'str'"}
WARNING:__main__:Present in both:
{"Comments contain 'nightly'",
 "Comments contain 'regression'",
 "First comment contains 'we'",
 'intermittent-failure in keywords',
 "status has ever been set to 'affected' and 'unaffected'=False"}
WARNING:__main__:Feature for model 'regression' changed between 2019-09-16 00:01:14+00:00 and 2019-09-17 14:35:27+00:00
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
{"First comment contains '__file_reference__'", "Comments contain 'str'"}
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
set()
WARNING:__main__:Present in both:
{"Comments contain 'nightly'",
 "Comments contain 'regression'",
 "First comment contains 'we'",
 'intermittent-failure in keywords',
 "status has ever been set to 'affected' and 'unaffected'=False"}
WARNING:__main__:Feature for model 'regression' changed between 2019-09-17 14:35:27+00:00 and 2019-09-18 18:29:55+00:00
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain 'nightly'"}
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
{"First comment contains '__file_reference__'"}
WARNING:__main__:Present in both:
{"Comments contain 'regression'",
 "First comment contains 'we'",
 'intermittent-failure in keywords',
 "status has ever been set to 'affected' and 'unaffected'=False"}
WARNING:__main__:Feature for model 'regression' changed between 2019-09-18 18:29:55+00:00 and 2019-09-24 21:59:18+00:00
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
{"First comment contains '__file_reference__'"}
WARNING:__main__:Feature only present at 2019-09-24 21:59:18+00:00:
{"Comments contain 'nightly'", "Comments contain '10'"}
WARNING:__main__:Present in both:
{"Comments contain 'regression'",
 "First comment contains 'we'",
 'intermittent-failure in keywords',
 "status has ever been set to 'affected' and 'unaffected'=False"}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-11 23:58:43+00:00 and 2019-09-12 08:15:25+00:00
WARNING:__main__:Feature only present at 2019-09-11 23:58:43+00:00:
set()
WARNING:__main__:Feature only present at 2019-09-12 08:15:25+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-12 11:26:14+00:00 and 2019-09-16 00:01:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
set()
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-16 00:01:14+00:00 and 2019-09-17 14:35:27+00:00
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
set()
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-17 14:35:27+00:00 and 2019-09-18 18:29:55+00:00
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
set()
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressor' changed between 2019-09-06 15:33:54+00:00 and 2019-09-07 00:06:33+00:00
WARNING:__main__:Feature only present at 2019-09-06 15:33:54+00:00:
{'# of times the directories were touched before (max 90 days)'}
WARNING:__main__:Feature only present at 2019-09-07 00:06:33+00:00:
{"Description contains 'commit'"}
WARNING:__main__:Present in both:
{'# of lines added',
 '# of lines added in tests',
 '# of times the directories were touched before (avg backout)',
 '# of times the directories were touched before (max backout)',
 '# of times the directories were touched before (max)'}
WARNING:__main__:Feature for model 'regressor' changed between 2019-09-07 00:06:33+00:00 and 2019-09-11 19:32:28+00:00
WARNING:__main__:Feature only present at 2019-09-07 00:06:33+00:00:
{"Description contains 'commit'"}
WARNING:__main__:Feature only present at 2019-09-11 19:32:28+00:00:
{'# of times the directories were touched before (max 90 days)'}
WARNING:__main__:Present in both:
{'# of lines added',
 '# of lines added in tests',
 '# of times the directories were touched before (avg backout)',
 '# of times the directories were touched before (max backout)',
 '# of times the directories were touched before (max)'}
WARNING:__main__:Feature for model 'regressor' changed between 2019-09-11 23:58:43+00:00 and 2019-09-12 08:15:25+00:00
WARNING:__main__:Feature only present at 2019-09-11 23:58:43+00:00:
{'# of times the directories were touched before (max 90 days)'}
WARNING:__main__:Feature only present at 2019-09-12 08:15:25+00:00:
{'# of times the files were touched before (sum 90 days)',
 "Description contains 'commit'"}
WARNING:__main__:Present in both:
{'# of lines added',
 '# of lines added in tests',
 '# of times the directories were touched before (avg backout)',
 '# of times the directories were touched before (max backout)',
 '# of times the directories were touched before (max)'}
WARNING:__main__:Feature for model 'regressor' changed between 2019-09-12 08:15:25+00:00 and 2019-09-12 11:26:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 08:15:25+00:00:
{'# of times the files were touched before (sum 90 days)',
 "Description contains 'commit'"}
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
set()
WARNING:__main__:Present in both:
{'# of lines added',
 '# of lines added in tests',
 '# of times the directories were touched before (avg backout)',
 '# of times the directories were touched before (max backout)',
 '# of times the directories were touched before (max)'}
WARNING:__main__:Feature for model 'regressor' changed between 2019-09-12 11:26:14+00:00 and 2019-09-16 00:01:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
set()
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
{'# of times the directories were touched before (max 90 days)'}
WARNING:__main__:Present in both:
{'# of lines added',
 '# of lines added in tests',
 '# of times the directories were touched before (avg backout)',
 '# of times the directories were touched before (max backout)',
 '# of times the directories were touched before (max)'}
WARNING:__main__:Feature for model 'regressor' changed between 2019-09-18 18:29:55+00:00 and 2019-09-24 21:59:18+00:00
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
{'# of times the directories were touched before (max 90 days)'}
WARNING:__main__:Feature only present at 2019-09-24 21:59:18+00:00:
{"Description contains 'commit'"}
WARNING:__main__:Present in both:
{'# of lines added',
 '# of lines added in tests',
 '# of times the directories were touched before (avg backout)',
 '# of times the directories were touched before (max backout)',
 '# of times the directories were touched before (max)'}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-06 15:33:54+00:00 and 2019-09-07 00:06:33+00:00
WARNING:__main__:Feature only present at 2019-09-06 15:33:54+00:00:
{"Comments contain 'attached'",
 "Comments contain 'bug'",
 "Comments contain 'build'",
 "Comments contain 'created'",
 "Comments contain 'first'",
 "Comments contain 'how'",
 "Comments contain 'on'",
 "Comments contain 'that'"}
WARNING:__main__:Feature only present at 2019-09-07 00:06:33+00:00:
{"Comments contain 'add'",
 "Comments contain 'an'",
 "Comments contain 'application'",
 "Comments contain 'are'",
 "Comments contain 'at'",
 "Comments contain 'attachment'",
 "Comments contain 'bit'",
 "Comments contain 'browser'",
 "Comments contain 'but'",
 "Comments contain 'could'",
 "Comments contain 'has'",
 "Comments contain 'information'",
 "Comments contain 'issue'",
 "Comments contain 'it'",
 "Comments contain 'like'",
 "Comments contain 'nightly'",
 "Comments contain 'not'",
 "Comments contain 'please'",
 "Comments contain 'reply'",
 "Comments contain 'reproduce'",
 "Comments contain 'revision'",
 "Comments contain 'should'",
 "Comments contain 'str'",
 "Comments contain 'ubuntu'",
 "Comments contain 'with'",
 "Title contains 'the'"}
WARNING:__main__:Present in both:
{"Comments contain '__url__'",
 "Comments contain 'as'",
 "Comments contain 'can'",
 "Comments contain 'expected'",
 "Comments contain 'fixed'",
 "Comments contain 'if'",
 "Comments contain 'in'",
 "Comments contain 'provide'",
 'Has Regression Range=yes'}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-07 00:06:33+00:00 and 2019-09-11 19:32:28+00:00
WARNING:__main__:Feature only present at 2019-09-07 00:06:33+00:00:
{"Comments contain '__url__'",
 "Comments contain 'an'",
 "Comments contain 'application'",
 "Comments contain 'at'",
 "Comments contain 'bit'",
 "Comments contain 'browser'",
 "Comments contain 'but'",
 "Comments contain 'could'",
 "Comments contain 'fixed'",
 "Comments contain 'if'",
 "Comments contain 'information'",
 "Comments contain 'issue'",
 "Comments contain 'it'",
 "Comments contain 'like'",
 "Comments contain 'nightly'",
 "Comments contain 'reply'",
 "Comments contain 'reproduce'",
 "Comments contain 'revision'",
 "Comments contain 'str'",
 "Comments contain 'ubuntu'",
 "Title contains 'the'"}
WARNING:__main__:Feature only present at 2019-09-11 19:32:28+00:00:
{"Comments contain '0a1'",
 "Comments contain '10'",
 "Comments contain 'been'",
 "Comments contain 'comment'",
 "Comments contain 'no'",
 "Comments contain 'problem'",
 "Comments contain 'process'",
 "Comments contain 'regression'",
 "Comments contain 'see'",
 "Comments contain 'that'",
 "Comments contain 'version'",
 "Comments contain 'you'"}
WARNING:__main__:Present in both:
{"Comments contain 'add'",
 "Comments contain 'are'",
 "Comments contain 'as'",
 "Comments contain 'attachment'",
 "Comments contain 'can'",
 "Comments contain 'expected'",
 "Comments contain 'has'",
 "Comments contain 'in'",
 "Comments contain 'not'",
 "Comments contain 'please'",
 "Comments contain 'provide'",
 "Comments contain 'should'",
 "Comments contain 'with'",
 'Has Regression Range=yes'}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-11 19:32:28+00:00 and 2019-09-11 23:58:43+00:00
WARNING:__main__:Feature only present at 2019-09-11 19:32:28+00:00:
{"Comments contain '0a1'",
 "Comments contain '10'",
 "Comments contain 'add'",
 "Comments contain 'as'",
 "Comments contain 'been'",
 "Comments contain 'can'",
 "Comments contain 'has'",
 "Comments contain 'in'",
 "Comments contain 'no'",
 "Comments contain 'not'",
 "Comments contain 'process'",
 "Comments contain 'see'",
 "Comments contain 'version'",
 "Comments contain 'with'",
 "Comments contain 'you'",
 'Has Regression Range=yes'}
WARNING:__main__:Feature only present at 2019-09-11 23:58:43+00:00:
{"Comments contain '14'",
 "Comments contain '__url__'",
 "Comments contain 'at'",
 "Comments contain 'code'",
 "Comments contain 'feature'",
 "Comments contain 'for'",
 "Comments contain 'if'",
 "Comments contain 'is'",
 "Comments contain 'just'",
 "Comments contain 'of'",
 "Comments contain 'on'",
 "Comments contain 'same'",
 "Comments contain 'screenshot'",
 "Comments contain 'still'",
 "Comments contain 'verified'"}
WARNING:__main__:Present in both:
{"Comments contain 'are'",
 "Comments contain 'attachment'",
 "Comments contain 'comment'",
 "Comments contain 'expected'",
 "Comments contain 'please'",
 "Comments contain 'problem'",
 "Comments contain 'provide'",
 "Comments contain 'regression'",
 "Comments contain 'should'",
 "Comments contain 'that'"}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-11 23:58:43+00:00 and 2019-09-12 08:15:25+00:00
WARNING:__main__:Feature only present at 2019-09-11 23:58:43+00:00:
{"Comments contain 'at'",
 "Comments contain 'on'",
 "Comments contain 'same'",
 "Comments contain 'still'",
 "Comments contain 'verified'"}
WARNING:__main__:Feature only present at 2019-09-12 08:15:25+00:00:
{"Comments contain 'as'"}
WARNING:__main__:Present in both:
{"Comments contain '14'",
 "Comments contain '__url__'",
 "Comments contain 'are'",
 "Comments contain 'attachment'",
 "Comments contain 'code'",
 "Comments contain 'comment'",
 "Comments contain 'expected'",
 "Comments contain 'feature'",
 "Comments contain 'for'",
 "Comments contain 'if'",
 "Comments contain 'is'",
 "Comments contain 'just'",
 "Comments contain 'of'",
 "Comments contain 'please'",
 "Comments contain 'problem'",
 "Comments contain 'provide'",
 "Comments contain 'regression'",
 "Comments contain 'screenshot'",
 "Comments contain 'should'",
 "Comments contain 'that'"}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-12 11:26:14+00:00 and 2019-09-16 00:01:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
{"Comments contain '14'",
 "Comments contain '__url__'",
 "Comments contain 'are'",
 "Comments contain 'feature'",
 "Comments contain 'if'",
 "Comments contain 'of'",
 "Comments contain 'problem'",
 "Comments contain 'screenshot'",
 "Comments contain 'that'"}
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
{"Comments contain '64'",
 "Comments contain 'created'",
 "Comments contain 'don'",
 "Comments contain 'in'",
 "Comments contain 'process'",
 "Comments contain 'reply'",
 "Comments contain 'reproduce'",
 "Comments contain 'str'",
 "Comments contain 'test'",
 "Comments contain 'the'",
 "Comments contain 'you'",
 'Has Regression Range=yes',
 'regression in keywords'}
WARNING:__main__:Present in both:
{"Comments contain 'as'",
 "Comments contain 'attachment'",
 "Comments contain 'code'",
 "Comments contain 'comment'",
 "Comments contain 'expected'",
 "Comments contain 'for'",
 "Comments contain 'is'",
 "Comments contain 'just'",
 "Comments contain 'please'",
 "Comments contain 'provide'",
 "Comments contain 'regression'",
 "Comments contain 'should'"}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-16 00:01:14+00:00 and 2019-09-17 14:35:27+00:00
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
{"Comments contain '64'",
 "Comments contain 'as'",
 "Comments contain 'code'",
 "Comments contain 'comment'",
 "Comments contain 'created'",
 "Comments contain 'don'",
 "Comments contain 'for'",
 "Comments contain 'in'",
 "Comments contain 'is'",
 "Comments contain 'process'",
 "Comments contain 'provide'",
 "Comments contain 'regression'",
 "Comments contain 'reply'",
 "Comments contain 'reproduce'",
 "Comments contain 'str'",
 "Comments contain 'test'",
 "Comments contain 'the'",
 'regression in keywords'}
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain '__code_reference_url__'",
 "Comments contain '__url__'",
 "Comments contain 'are'",
 "Comments contain 'bp'",
 "Comments contain 'causing'",
 "Comments contain 'let'",
 "Comments contain 'not'",
 "Comments contain 'still'",
 "Comments contain 'version'",
 "Comments contain 'with'"}
WARNING:__main__:Present in both:
{"Comments contain 'attachment'",
 "Comments contain 'expected'",
 "Comments contain 'just'",
 "Comments contain 'please'",
 "Comments contain 'should'",
 "Comments contain 'you'",
 'Has Regression Range=yes'}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-17 14:35:27+00:00 and 2019-09-18 18:29:55+00:00
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain '__url__'",
 "Comments contain 'are'",
 "Comments contain 'bp'",
 "Comments contain 'causing'",
 "Comments contain 'expected'",
 "Comments contain 'let'",
 "Comments contain 'not'",
 "Comments contain 'should'",
 "Comments contain 'still'",
 "Comments contain 'version'"}
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
{"Comments contain 'bug'",
 "Comments contain 'can'",
 "Comments contain 'file'",
 "Comments contain 'issue'",
 "Comments contain 'nightly'",
 "Comments contain 'open'",
 "Comments contain 'other'",
 "Comments contain 'provide'",
 "Comments contain 'result'",
 "Comments contain 'tab'",
 "Comments contain 'the'",
 "Comments contain 'windows'",
 "Title contains 'is'"}
WARNING:__main__:Present in both:
{"Comments contain '__code_reference_url__'",
 "Comments contain 'attachment'",
 "Comments contain 'just'",
 "Comments contain 'please'",
 "Comments contain 'with'",
 "Comments contain 'you'",
 'Has Regression Range=yes'}
WARNING:__main__:Feature for model 'stepstoreproduce' changed between 2019-09-18 18:29:55+00:00 and 2019-09-24 21:59:18+00:00
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
{"Comments contain '__code_reference_url__'",
 "Comments contain 'file'",
 "Comments contain 'issue'",
 "Comments contain 'just'",
 "Comments contain 'nightly'",
 "Comments contain 'open'",
 "Comments contain 'other'",
 "Comments contain 'provide'",
 "Comments contain 'tab'",
 "Comments contain 'the'",
 "Comments contain 'windows'",
 "Comments contain 'with'",
 "Title contains 'is'"}
WARNING:__main__:Feature only present at 2019-09-24 21:59:18+00:00:
{"Comments contain '0a1'",
 "Comments contain 'as'",
 "Comments contain 'created'",
 "Comments contain 'expected'",
 "Comments contain 'for'",
 "Comments contain 'how'",
 "Comments contain 'if'",
 "Comments contain 'it'",
 "Comments contain 'new'",
 "Comments contain 'not'",
 "Comments contain 'same'",
 "Comments contain 'should'"}
WARNING:__main__:Present in both:
{"Comments contain 'attachment'",
 "Comments contain 'bug'",
 "Comments contain 'can'",
 "Comments contain 'please'",
 "Comments contain 'result'",
 "Comments contain 'you'",
 'Has Regression Range=yes'}

Some features comes and leave in a cycle, for example in the regressionrange model:

WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-11 23:58:43+00:00 and 2019-09-12 08:15:25+00:00
WARNING:__main__:Feature only present at 2019-09-11 23:58:43+00:00:
set()
WARNING:__main__:Feature only present at 2019-09-12 08:15:25+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-12 11:26:14+00:00 and 2019-09-16 00:01:14+00:00
WARNING:__main__:Feature only present at 2019-09-12 11:26:14+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
set()
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-16 00:01:14+00:00 and 2019-09-17 14:35:27+00:00
WARNING:__main__:Feature only present at 2019-09-16 00:01:14+00:00:
set()
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}
WARNING:__main__:Feature for model 'regressionrange' changed between 2019-09-17 14:35:27+00:00 and 2019-09-18 18:29:55+00:00
WARNING:__main__:Feature only present at 2019-09-17 14:35:27+00:00:
{"Comments contain 'good'"}
WARNING:__main__:Feature only present at 2019-09-18 18:29:55+00:00:
set()
WARNING:__main__:Present in both:
{'# of landing comments',
 "Comments contain '__code_reference_url__'",
 "Comments contain 'revision'",
 'Has STR=yes'}

It seems mostly to be features coming or leaving the top features in the report.

Lothiraldan avatar Sep 26 '19 19:09 Lothiraldan

stepstoreproduce and regressionrange are quite unstable as they aren't really ready for production yet, so that's expected. Maybe we should have a way to define models to ignore when checking (same for the metrics).

Anyway, let's log all feature changes in the past, but only warn and fail the task when there is a difference between the previous and the current.

Question: what is that "set()" feature?

marco-c avatar Sep 28 '19 12:09 marco-c

Will need to be rebased, but I will keep this open in case we want to pick it up again.

I linked the PR to issue #824 in case we want to pick it up again. For now, closing for lack of updates, feel free to reopen if you still intend to work on this.

suhaibmujahid avatar Mar 10 '23 13:03 suhaibmujahid