lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

Homogeneize logging system

Open clefourrier opened this issue 1 year ago • 14 comments

Make it so both model types can run tensorboard evals.

This should be good to go, but will need to wait for an update in huggingface_hub before merging, as the lib stops the job if ŧensorboard is installed when it's not needed (see Slack for discussions), and cc @Wauplin :)

Closes #118

clefourrier avatar Apr 08 '24 16:04 clefourrier

Opened https://github.com/huggingface/huggingface_hub/pull/2205 for the huggingface_hub part :)

Wauplin avatar Apr 09 '24 07:04 Wauplin

Thanks :) In which huggingface_hub release will this be included?

clefourrier avatar Apr 09 '24 13:04 clefourrier

Thanks :) In which huggingface_hub release will this be included?

0.23.x

Not sure when it'll be though (not before ~10 days I think, given last release was recent and not much has been done since then).

Wauplin avatar Apr 09 '24 13:04 Wauplin

A possible solution if you want a quick fix is to install from source. Not the best if the lib' is used "in prod".

Wauplin avatar Apr 09 '24 13:04 Wauplin

We'll wait for your release then! :) (It's not used in prod but we want to keep the pytoml clean-ish)

clefourrier avatar Apr 09 '24 13:04 clefourrier

Understandable!

Wauplin avatar Apr 09 '24 13:04 Wauplin

@Wauplin is your release coming soon? :)

clefourrier avatar Apr 20 '24 15:04 clefourrier

Not at short term actually (with ~10 days). I haven't pushed much changes to huggingface_hub since this fix. You would prefer something more short-term I guess?

Wauplin avatar Apr 22 '24 08:04 Wauplin

If it were possible to push a patch for this it would be great, but else we'll leave it hanging - it's a part of the code we're not usually touching a lot so it won't be a hassle to merge in the future.

clefourrier avatar Apr 22 '24 08:04 clefourrier

I'd prefer not to do a patch release for that as other "import-related" stuff has been fixed as well in huggingface_hub and cherry-picking commits would be hazardous (and taking from main would not be a patch release).

Another solution is to rely on the main branch for 2 weeks and I can take care of pinning again the version once huggingface_hub 0.23 is out.

Wauplin avatar Apr 22 '24 08:04 Wauplin

Nope we'll wait - it's better for us to only rely on releases to avoid unexpected silent failures - but thanks :)

clefourrier avatar Apr 22 '24 08:04 clefourrier

hey @Wauplin ! any news on the state of the release ? :)

NathanHB avatar May 12 '24 11:05 NathanHB

Sorry sorry, forgot to let you know before going on holidays :see_no_evil: Huggingface_hub 0.23 has been shipped 3 weeks ago :grimacing: https://github.com/huggingface/huggingface_hub/releases/tag/v0.23.0

Wauplin avatar May 21 '24 13:05 Wauplin

@NathanHB tested it and it should be good and mergeable when you've got the time :)

clefourrier avatar May 22 '24 15:05 clefourrier