DataCollatorWithFlattening is incompatible with non - list input ids

[alex-hh]

System Info

latest transformers

Who can help?

@ArthurZucker

Information

• [ ] The official example scripts
• [x] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [x] My own task or dataset (give details below)

Reproduction

from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
example = tokenizer("A test sentence", return_tensors="pt")
example = {k: v.flatten() for k, v in tensor_example.items()}
collator([example]*2)

Expected behavior

Collator should work with all output types supported by tokenizer.

[gaurangk19]

Hi! I am planning on working on this under Hacktoberfest 2024. Can you assign me this issue? I hope I am able to solve this

[ArthurZucker]

Hey! We do not assign issues, feel free to open a PR 🤗

[sudhanshu746]

@ArthurZucker , Would you be able to review and merge this request? Thank you.