Example for LMGradientAttribution is missing.

[saxenarohit]

📚 Documentation

This is in reference to the tutorial page below. https://captum.ai/tutorials/Llama2_LLM_Attribution

I could not find the example for LLMGradientAttribution for LLAMA2.

Any help on this will be appreciated.

Thanks

[aobo-y]

@saxenarohit thanks for reminding us. We will add it soon

[Dongximing]

Hi aobo, if you are busy, could you tell me which model layer need I take as a parameters in the LayerIntegratedGradients ?

[aobo-y]

hi @Dongximing , it should be the embedding layer of your model. As a token is discrete, its backpropagate gradient stop at its embedding. For Llama2, it would something like the following

emb_layer = model.get_submodule("model.embed_tokens")
lig = LayerIntegratedGradients(model, emb_layer)

[Dongximing]

Thanks, I saw the result, and analysis the code, the final results are computed on log_softmax. and is that means if a contribution in this way, -10,20,-20. the token_1 and token_2 are both important? or we need "abs()" to eval the important of tokens?

[qingyuanxingsi]

Any tutorial update? @aobo-y

[craymichael]

Hi, we have finally added this in https://github.com/pytorch/captum/commit/e01c07b741be53bbf77d425725088cc0dd430005. Thanks again for bringing this to our attention!