Orion icon indicating copy to clipboard operation
Orion copied to clipboard

Issues with the original code/paper

Open AlexanderGeiger opened this issue 6 years ago • 0 comments

There are a few things in the original NASA project/paper that are a bit confusing, so I will list them here for your information and future reference:

  • The labels for the anomalous regions of the channels in the labeled_anomalies.csv file seem to be off by some number (presumably 260). In the original implementation this offset is cancelled out again, but the anomaly regions that are reported in the file are not correct per se and we have to take care of that in our implementation. This should already be reflected in the anomalies data set we have in our AWS bucket.
  • The F0.5 scores reported in the paper seem to be calculated incorrectly (their reported Precision and Recall scores should give a higher F0.5 score)
  • If we use the original code and retrain the models, we do not reach the same scores, but we are somewhere close (reported true F0.5 score is 0.859, while we reach 0.802. If the pre-trained models are taken, the score is quite similar to the paper).

AlexanderGeiger avatar Jun 27 '19 20:06 AlexanderGeiger