inference icon indicating copy to clipboard operation
inference copied to clipboard

Submission checker forces to have “gptj-99” and “gpt-99.9” in the code dir

Open szutenberg opened this issue 1 year ago • 1 comments

It ends up that we have even three identical directories (see NVIDIA submission) which does not help with the review and understanding the repo by someone not involved in the WG.

Solution: there should be one directory per model

if needed, the submitter can create 99 & 99.9 directories

Now:

  • gptj-99.9
  • gptj-99
  • gptj

After:

  • gptj

After - optionally, directories per accuracy category:

  • gptj/99.9
  • gptj/99

szutenberg avatar Apr 09 '24 15:04 szutenberg

I think we can discuss further on this. The problem with the proposal is that we no longer have the same model names inside the results/measurements directory (like this ) and the ones inside the code directory. Also to avoid duplicates in the code directory we can have softlinks as done here though this can be tricky on some filesystems.

arjunsuresh avatar Apr 10 '24 16:04 arjunsuresh