verifiers icon indicating copy to clipboard operation
verifiers copied to clipboard

feat: add response format handling in judge rubric

Open lakshyaag opened this issue 5 months ago • 4 comments

Description

From #331 This PR adds response_format to the default judge rubric parameters. On specifying a OpenAI-compatible ResponseFormat, the request will use the .parse() method in the OpenAI SDK.

Type of Change

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Documentation update
  • [ ] Test improvement

Testing

  • [x] All existing tests pass
  • [x] New tests have been added to cover the changes
  • [x] Tests have been run locally with uv run pytest

Test Coverage

  • Current coverage: ___%
  • Coverage after changes: ___%

Checklist

  • [x] My code follows the style guidelines of this project
  • [x] I have performed a self-review of my own code
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [x] My changes generate no new warnings
  • [ ] Any dependent changes have been merged and published

Additional Notes

lakshyaag avatar Sep 16 '25 05:09 lakshyaag

It would also be nice to be able to use response_format/structured responses in the user_client, or is this better handled with tool calling?

wcummings avatar Sep 17 '25 16:09 wcummings

@lakshyaag Can this not already be done by passing response_format via sampling_args? Would rather have that be the all-in-one route for expressing additional configurations rather than explicitly adding each one.

willccbb avatar Sep 18 '25 18:09 willccbb

@willccbb I'll look into it - should be able to support it.

lakshyaag avatar Sep 19 '25 19:09 lakshyaag

@cursoragent look into it

lakshyaag avatar Sep 20 '25 00:09 lakshyaag