HiLab-git
One issue about the loss function
Thanks for sharing the code of the wonderful work. I have a problem with the loss function. In section 2.2, it introduces the loss function of knowledge distillation, which is used to obtain the high entropy. In section 2.3, it introduces the loss function of uncertainty minimization, which is used to obtain the low entropy. Will there be a conflict between these two loss functions? I hope to receive your reply as soon as possible.
The experimental results demonstrate that these two loss functions are not contradictory. Regarding the issue you mentioned, I believe that the loss function based on Knowledge Distillation (KD) does not increase entropy.
In section 2.2, you said "Note that T = 1 corresponds to a standard Softmax function, and a larger T value leads to a softer probability distribution with higher entropy. ". In the experiment, you set T=10. Could you explain how the knowledge distillation works in principle? (rather than just answering this question from the perspective of experimental results) Thanks.
This entropy refers to an increase in the entropy of the softened distribution, but it doesn't necessarily imply that the goal of this loss is to increase the entropy of the prediction results.
abcxubu @.***> 于2023年8月25日周五 16:42写道：
In section 2.2, you said "Note that T = 1 corresponds to a standard Softmax function, and a larger T value leads to a softer probability distribution with higher entropy. ". In the experiment, you set T=10. Could you explain how the knowledge distillation works in principle? (rather than just answering this question from the perspective of experimental results) Thanks.
— Reply to this email directly, view it on GitHub https://github.com/HiLab-git/CDMA/issues/2#issuecomment-1692996363, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWZLYV33EPQEHLCVKTG2LWDXXBQO7ANCNFSM6AAAAAA3ZLUFEQ . You are receiving this because you commented.Message ID: @.***>