macaodha
Unrealistic probability in summary
Probability is scaled to add up to 100% in the summary. But 100% of rubbish will only be rubbish. Need a realistic probability to avoid wasting time looking at rubbish files.
I am currently using the average probability in the summary, but I am looking to improve on this for multiple species in one file. Hopefully, GUI will help with multiple species in the same file, when it is fully developed.
Hi @markgloverswaybtinternetcom.
Thanks for the feedback!
When you say "realistic probability", could you clarify a bit what you'd ideally like to see?
I think the model outputs det_prob and class_prob are somewhat of a misnomer; they shouldn't be interpreted as actual probabilities, but more as confidence scores. As is common with deep learning models, these scores often don't align with the intuitive idea of the probability that a bat is actually present.
In practice, it's best to calibrate these scores using validated detections from your own data to get closer to a "true" probability. This paper (https://link.springer.com/article/10.1007/s10336-024-02144-5) describes one approach, though as the authors point out, calibration is often very dataset-specific (affected by factors like the device, site, and local conditions), so what works for us might not work for you.
How best to use the model outputs for different monitoring goals is still an open question. I would be keen to hear more about your use case; what are you using the outputs for, and what would you ideally like to see happen?
Cheers, Santiago
Hi Below is the what I was seeing. I think the problem is the calls at at the upper limit for Pips, this why BatDetect2 is making this mistake. But Alcathoes are rare.
I have tried the BTO pipeline which seems to have the opposite problem only recognising a few perfect calls (which is good for quality), whereas BatDetect2 tries to recognise everything. I prefer a more interactive approach that BatDetect2 is capable of How you label calls for training probably has a big effect e.g. how big the boxes are. Regards Mark Glover
------ Original Message ------ From "Santiago Martinez Balvanera" @.> To "macaodha/batdetect2" @.> Cc "markgloverswaybtinternetcom" @.>; "Mention" @.> Date 02/06/2025 19:03:52 Subject Re: [macaodha/batdetect2] Unrealistic probability in summary (Issue #46)
mbsantiago left a comment (macaodha/batdetect2#46) https://github.com/macaodha/batdetect2/issues/46#issuecomment-2931853893 Hi @markgloverswaybtinternetcom https://github.com/markgloverswaybtinternetcom.
Thanks for the feedback!
When you say "realistic probability", could you clarify a bit what you'd ideally like to see?
I think the model outputs det_prob and class_prob are somewhat of a misnomer; they shouldn't be interpreted as actual probabilities, but more as confidence scores. As is common with deep learning models, these scores often don't align with the intuitive idea of the probability that a bat is actually present.
In practice, it's best to calibrate these scores using validated detections from your own data to get closer to a "true" probability. This paper (https://link.springer.com/article/10.1007/s10336-024-02144-5) describes one approach, though as the authors point out, calibration is often very dataset-specific (affected by factors like the device, site, and local conditions), so what works for us might not work for you.
How best to use the model outputs for different monitoring goals is still an open question. I would be keen to hear more about your use case; what are you using the outputs for, and what would you ideally like to see happen?
Cheers, Santiago
— Reply to this email directly, view it on GitHub https://github.com/macaodha/batdetect2/issues/46#issuecomment-2931853893, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZP7OYBDO6GTMXFFKEFKZ4L3BSGYRAVCNFSM6AAAAAB23OP2LWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMZRHA2TGOBZGM. You are receiving this because you were mentioned.Message ID: @.***>
-- This email has been checked for viruses by Avast antivirus software. www.avast.com