deepset-ai
feat: LLM-based evaluators return meta info from OpenAI
Related Issues
- fixes #7905
Proposed Changes:
pass the meta entry from openai api to the result for all the LLM-based evaluators, containing the following fields:
- model
- prompt tokens
- answer tokens
- total tokens
How did you test it?
manual veritifcation + run local tests + end2endtests
Checklist
- I have read the contributors guidelines and the code of conduct
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the conventional commit types for my PR title:
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:. - I documented my code
- I ran pre-commit hooks and fixed any issue
Pull Request Test Coverage Report for Build 9711234163
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
Pull Request Test Coverage Report for Build 9711238547
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
Pull Request Test Coverage Report for Build 9711837271
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
Pull Request Test Coverage Report for Build 9712238785
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
NOTE: It's ready for review only an issue with a bug in pylint:
- https://github.com/deepset-ai/haystack/actions/runs/9712238785/job/26806852617?pr=7947
Waiting for the new release of pylint (hopefully soon) to fix it.
Pull Request Test Coverage Report for Build 9713035923
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
Seems like the last commit caused the tests to fail with different names. Otherwise looks good!
Pull Request Test Coverage Report for Build 9757781468
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
- For more information on this, see Tracking coverage changes with pull request builds.
- To avoid this issue with future PRs, see these Recommended CI Configurations.
- For a quick fix, rebase this PR at GitHub. Your next report should be accurate.
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
Pull Request Test Coverage Report for Build 9758083139
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
- For more information on this, see Tracking coverage changes with pull request builds.
- To avoid this issue with future PRs, see these Recommended CI Configurations.
- For a quick fix, rebase this PR at GitHub. Your next report should be accurate.
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |
💛 - Coveralls
Pull Request Test Coverage Report for Build 9758257289
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
- For more information on this, see Tracking coverage changes with pull request builds.
- To avoid this issue with future PRs, see these Recommended CI Configurations.
- For a quick fix, rebase this PR at GitHub. Your next report should be accurate.
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- 6 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-0.009%) to 89.968%
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| components/evaluators/llm_evaluator.py | 6 | 94.74% |
| <!-- | Total: | 6 |
| Totals | |
|---|---|
| Change from base Build 9710252897: | -0.009% |
| Covered Lines: | 6744 |
| Relevant Lines: | 7496 |