feat: add response format handling in judge rubric
Description
From #331
This PR adds response_format to the default judge rubric parameters. On specifying a OpenAI-compatible ResponseFormat, the request will use the .parse() method in the OpenAI SDK.
Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation update
- [ ] Test improvement
Testing
- [x] All existing tests pass
- [x] New tests have been added to cover the changes
- [x] Tests have been run locally with
uv run pytest
Test Coverage
- Current coverage: ___%
- Coverage after changes: ___%
Checklist
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] Any dependent changes have been merged and published
Additional Notes
It would also be nice to be able to use response_format/structured responses in the user_client, or is this better handled with tool calling?
@lakshyaag Can this not already be done by passing response_format via sampling_args? Would rather have that be the all-in-one route for expressing additional configurations rather than explicitly adding each one.
@willccbb I'll look into it - should be able to support it.
@cursoragent look into it