LLMeBench icon indicating copy to clipboard operation
LLMeBench copied to clipboard

Extended experiment configuration backup

Open MaramHasanain opened this issue 2 years ago • 0 comments

As part of the output of a benchmarking experiment, we should write the full configuration used to file. This can be useful to version experiments (for reproducibility). Example configs to maintain:

  • Running time and date of the experiment
  • Train and test dataset file names and the name of dataset script used to load them.
  • Task script name
  • Model
  • Base Prompt used
  • Learning setup (0-shot, few-shot, etc) and details about the setup (e.g., method used to select few shots)

"Might save all of this as a pickled object"

MaramHasanain avatar Sep 11 '23 05:09 MaramHasanain