Hunar Batra

icon indicating a website link hunarbatra.com

icon company University of Oxford icon location Oxford, UK icon location DPhil/MSc CS, University of Oxford

Results 1 comments of Hunar Batra

Phi 3 Mini 128K leads to Tokenization Mismatch

Faced the same problem! I've updated the phi-3 preprocessor as follows: ```python def preprocess_phi3( sources, tokenizer: transformers.PreTrainedTokenizer, has_image: bool = False ) -> Dict: conv = conversation_lib.conv_templates["phi3-instruct"].copy() roles = {"human":...