adk-python icon indicating copy to clipboard operation
adk-python copied to clipboard

tool_trajectory_avg_score class doesn't accept string values for `match_type`

Open alexhermida opened this issue 2 months ago • 0 comments

Describe the bug The new implementation in the ToolTrajectoryCriterion class doesn't accept string values for match_type (e.g., "ANY_ORDER") causing a Pydantic validation error. The enum expects integer values (0, 1, or 2) but in the documentation shows string values. It works with integers.

To Reproduce Create an evaluation configuration with a string value for match_type:

from google.adk.evaluation.eval_metrics import EvalMetric, ToolTrajectoryCriterion, PrebuiltMetrics

eval_metric = EvalMetric(
    threshold=0.5,
    metric_name=PrebuiltMetrics.TOOL_TRAJECTORY_AVG_SCORE.value,
    criterion={
        "threshold": 0.1,
        "match_type": "ANY_ORDER"  # Using string as documented
    }
)

Error:

  pydantic_core._pydantic_core.ValidationError: 1 validation error for ToolTrajectoryCriterion
  match_type
    Input should be 0, 1 or 2 [type=enum, input_value='ANY_ORDER', input_type=str]
      For further information visit https://errors.pydantic.dev/2.12/v/enum

Expected behavior The criterion should accept string values ("EXACT", "IN_ORDER", "ANY_ORDER") as shown in the ADK documentation.

Desktop:

  • OS: macOS
  • Python version: 3.13.0
  • ADK version: 1.19.0

Root Cause The MatchType enum in src/google/adk/evaluation/eval_metrics.py uses integer values (0, 1, 2), but Pydantic v2 doesn't automatically convert string enum names to enum values without an explicit validator.

Proposed Solution Add a Pydantic field_validator to the ToolTrajectoryCriterion.match_type field that converts string enum names to their corresponding enum values if we want to keep compatibility with integer values.

I've already tested a fix locally and can open a PR if that would be helpful, with comprehensive test coverage of course. Let me know if you'd like me to contribute the fix.

Thanks for all hard work!

alexhermida avatar Nov 25 '25 16:11 alexhermida