Karim Foda

Add stop sequence to text generation pipeline

10

# What does this PR do? As per the conversation in https://github.com/huggingface/transformers/issues/17562, creating this draft PR to add a stop_sequence option to text generation pipelines. Fixes # (issue) ## Before...

Flax Remat for LongT5

6

# What does this PR do? Following on from the discussion in https://github.com/huggingface/transformers/issues/17399 this PR adds remat (flax's gradient checkpointing) to LongT5. Most of the code is copied from https://github.com/huggingface/transformers/pull/17843/files...

Add hallucination filter in generate()

4

### Feature request Adding a filter of some sorts in the generate function to prevent x number of words from outside of the input from appearing in a generated text....

No local packages or working download links found for tokenizers==0.12.1.dev0

5

I get the following Error when installing tokenisers from source (as I'm on Macbook's M1 so can't install using pip I believe): ``` Copying rust artifact from /Users/karimfoda/Documents/STUDIES/PYTHON/SHORTFORM/tokenizers/bindings/python/target/release/libtokenizers.dylib to build/lib.macosx-11.1-arm64-cpython-310/tokenizers/tokenizers.cpython-310-darwin.so...

Add hallucination filter

2

# What does this PR do? As per the discussions in https://github.com/huggingface/transformers/issues/18354, this PR aims to add a `HallucinationPenaltyLogitsProcessor` that takes in a `hallucination_penalty` that is applied to any tokens...

Compatibility Issues with MacBook M1 due to dependency on Tokenizers < 0.13.0

1

Hello, I was struggling to get this library loaded on an M1 MacBooks because the tokenizers library could not be automatically built for M1s. Tokenisers 0.13.0 [now supports this](https://github.com/huggingface/tokenizers/issues/712#issuecomment-1259638471) but...

"text2text-generation" pipeline fails when setting return_dict_in_generate=True

2

### System Info - `transformers` version: 4.23.1 - Platform: macOS-12.5.1-arm64-arm-64bit - Python version: 3.9.4 - Huggingface_hub version: 0.10.1 - PyTorch version (GPU?): 1.12.1 (False) - Tensorflow version (GPU?): 2.10.0 (True)...

Add BART-LS

5

### Model description BART-LS (Long Bart), presented in this [paper](https://arxiv.org/pdf/2209.10052.pdf), establishes a new SOTA on a number of NLP tasks and long form datasets. It uses pooling-augmented block-wise attention and...

Hi Team, Thanks for contributing this amazing dataset and open sourcing it. I had a question regarding usage of the data: The Legal note in the `README.md` file says: _"You...

### Bug https://discord.com/channels/1120750674595024897/1283483486182969465 Currently 6.12.3 does not have the fixed reconnection logic introduced in 7.4.0. This means that when running on testnet miners and validators get the attached error: ![Screenshot_2024-09-11_at_09...