tool_calling_api icon indicating copy to clipboard operation
tool_calling_api copied to clipboard

Add captum, inspect, and llm as judges

Open Shuyib opened this issue 4 months ago • 0 comments

Feature Request

Please add captum, inspect, and llm as judge components in the project. This will allow for broader evaluation capabilities and integration with advanced model analysis tools.

Requested features to add:

  • captum for model interpretability of a encoder-decoder model
  • inspect review PII detection, toxicity and prompt injection
  • llm as a judge to review the logs incase of any issues

Motivation

Adding these judges will enhance model interpretability, debugging, and assessment by leveraging specialized tools.

Expected Outcome

  • All three judges (captum, inspect, llm) are available and can be called as part of the evaluation pipeline.
  • Documentation is updated to reflect their usage.

Shuyib avatar Sep 10 '25 04:09 Shuyib