DeepSpeed
DeepSpeed copied to clipboard
microsoft
[BUG] Non-Deterministic Model Responses when the Input Prompt Order Changes
Describe the bug When using deepspeed to run inference using a llama2 model, I see different model responses when the order of the prompts passing into the model changes. But if the order of prompts remain the same, the responses are the same. Is it an expected behaviour? I don't tend to expect the order to change model responses. Please let me know if I am wrong.
To Reproduce Steps to reproduce the behavior:
- Simple inference script to reproduce
import os
import torch
import deepspeed
import time
from tqdm import tqdm
import jsonlines
from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
set_seed(42)
local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
hf_token = '<replace with your huggingface token>'
max_tokens = 4096
batch_size = 1
input_list = [
"Can you tell me about the important people in your life?",
"What have been some of the happiest moments in your life? The saddest?",
"Who has been the biggest influence on your life? What lessons did that person or those people teach you?",
"Can you tell me about a moment when a person’s kindness made a difference in your life?",
"What are the most important lessons you’ve learned in life?",
"What is your earliest memory?"
]
# input_list = [
# "Who has been the biggest influence on your life? What lessons did that person or those people teach you?",
# "Can you tell me about the important people in your life?",
# "Can you tell me about a moment when a person’s kindness made a difference in your life?",
# "What have been some of the happiest moments in your life? The saddest?",
# "What is your earliest memory?",
# "What are the most important lessons you’ve learned in life?",
# ]
input_list = [input_list[x:x+batch_size] for x in range(0, len(input_list), batch_size)]
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Llama-2-7b-chat-hf",
use_fast=True,
token=hf_token,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-chat-hf",
torch_dtype=torch.float16,
device_map="cpu",
token=hf_token,
)
tokenizer.add_special_tokens({"pad_token": '[PAD]'})
model.resize_token_embeddings(len(tokenizer))
model.config.pad_token_id = tokenizer.pad_token_id
model.eval()
model = deepspeed.init_inference(
model=model,
mp_size=world_size,
dtype=torch.float16,
replace_with_kernel_inject=False,
max_tokens=max_tokens,
)
output_list = []
for inputs in tqdm(input_list):
torch.cuda.synchronize()
input_tokens = tokenizer.batch_encode_plus(
[f"[INST]{inp}[/INST]" for inp in inputs],
return_tensors="pt",
padding="longest",
truncation=True,
max_length=4096
)
for t in input_tokens:
if torch.is_tensor(input_tokens[t]):
input_tokens[t] = input_tokens[t].to(local_rank)
outputs = model.generate(**input_tokens, max_length=4096, do_sample=True)
response_outputs = []
for i, o in zip(input_tokens["input_ids"], outputs):
o = o[len(i):]
response_outputs.append(o)
response_outputs = tokenizer.batch_decode(
response_outputs,
skip_special_tokens=True,
clean_up_tokenization_spaces=True
)
response_outputs = [i.strip() for i in response_outputs]
output_list += response_outputs
torch.cuda.synchronize()
torch.cuda.empty_cache()
if local_rank==0:
print(response_outputs)
#save
if local_rank==0:
disk_output = []
for q, o in zip(input_list, output_list):
disk_output.append({"prompt": q, "output": o})
with jsonlines.open('./run3.jsonl', 'w') as writer:
writer.write_all(disk_output)
-
What packages are required and their versions jsonlines==3.1.0 torch==2.2.2+cu121 deepspeed==0.12.6 accelerate==0.30.1 Cuda driver 12.1
-
How to run the script
deepspeed --num_gpus 2 script.py
I run this on 2xT4s.
As you can see, there are 2 input_list in the script. I run the script for each input_list. The only thing that changed is the order of prompts passing to the model for response generation. I see some slight response differences between the 2 runs. However, if I keep order constant and run for the same input_list twice, the responses are exactly the same.
Expected behavior I would expect the order to not impact the responses. The responses should be somewhat deterministic as set_seed is set.
ds_report output
[2024-10-08 19:20:14,676] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2
[WARNING] using untested triton version (2.2.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/envs/llama2env/lib/python3.8/site-packages/torch']
torch version .................... 2.2.2+cu121
deepspeed install path ........... ['/opt/conda/envs/llama2env/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.12.6, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.1, cuda 12.1
shared memory (/dev/shm) size .... 25.55 GB
System info (please complete the following information):
- OS: Linux Debian 10
- GPU count and types: 2xT4s
- (if applicable) Hugging Face Transformers/Accelerate/etc. versions: Please see above
- Python version 3.8.15
- Any other relevant info about your setup Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU @ 2.20GHz Stepping: 0 CPU MHz: 2199.998 BogoMIPS: 4399.99 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 56320K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities
Docker context Are you using a specific docker image that you can share? No
Model Output
| prompt | output_x | output_y | |
|---|---|---|---|
| 0 | ['Who has been the biggest influence on your life? What lessons did that person or those people teach you?'] | My parents have been the biggest influence on my life. They have taught me many valuable lessons that have shaped me into the person I am today. Here are some of the most important lessons they have taught me:\n\n1. The importance of hard work and perseverance: My parents have always emphasized the importance of working hard and persevering through challenges. They have taught me that success rarely comes overnight, and that it takes dedication, determination, and persistence to achieve my goals.\n2. The value of education: My parents have always placed a strong emphasis on education, and have taught me that it is the key to unlocking opportunities and achieving my full potential. They have instilled in me a love of learning, and have encouraged me to pursue my academic interests and passions.\n3. The power of kindness and compassion: My parents have taught me the importance of treating others with kindness and compassion. They have shown me that small acts of kindness can go a long way in making a positive impact on someone's life, and have encouraged me to be a force for good in the world.\n4. The importance of family: My parents have taught me the value of family and the importance of prioritizing relationships. They have shown me that a strong support system is essential for navigating life's challenges, and have encouraged me to cherish and nurture my relationships with loved ones.\n5. The value of resilience: My parents have taught me that setbacks and failures are a natural part of life, and that it is important to bounce back from these challenges with resilience and determination. They have shown me that it is possible to overcome obstacles and emerge stronger and more resilient than before.\n6. The importance of self-care: My parents have taught me the importance of taking care of myself, both physically and mentally. They have shown me that self-care is essential for maintaining my health and well-being, and have encouraged me to prioritize activities that nurture my mind, body, and spirit.\n7. The value of gratitude: My parents have taught me the importance of gratitude and appreciation. They have shown me that it is essential to be thankful for what I have, and to express gratitude to those who have helped me along the way.\n8. The importance of giving back: My parents have taught me the importance of giving back to the community and helping others. They have shown me that it is essential to use my talents and resources to make a positive impact on the world, and to contribute to the greater good.\n\nOverall, my parents have taught me the importance of hard work, education, kindness, family, resilience, self-care, gratitude, and giving back. These lessons have had a profound impact on my life, and have helped me to become the person I am today. | My parents have been the biggest influence on my life. They have taught me so many valuable lessons that have shaped me into the person I am today. Here are some of the most important lessons they have taught me:\n\n1. Hard work and perseverance: My parents have always emphasized the importance of hard work and perseverance. They have taught me that nothing in life comes easily, and that in order to achieve my goals, I need to be willing to put in the effort and time required to succeed.\n2. Responsibility and accountability: My parents have taught me the importance of taking responsibility for my actions and being accountable for my mistakes. They have shown me that owning up to my mistakes and learning from them is essential for personal growth and development.\n3. Empathy and compassion: My parents have taught me to be empathetic and compassionate towards others. They have shown me that treating others with kindness and respect is essential for building strong relationships and creating a positive impact in the world.\n4. Self-discipline and self-control: My parents have taught me the importance of self-discipline and self-control. They have shown me that in order to achieve my goals, I need to be able to control my emotions and impulses, and stay focused on what is important.\n5. Gratitude and appreciation: My parents have taught me to be grateful and appreciative for what I have. They have shown me that it is important to recognize and appreciate the good things in life, and to never take anything for granted.\n6. Forgiveness and letting go: My parents have taught me the importance of forgiveness and letting go. They have shown me that holding onto grudges and resentment only causes harm and that it is essential to forgive and move on in order to heal and move forward.\n7. Self-awareness and self-reflection: My parents have taught me the importance of self-awareness and self-reflection. They have shown me that it is essential to understand my own strengths, weaknesses, and motivations in order to make informed decisions and live a fulfilling life.\n8. The value of education: My parents have taught me the value of education and the importance of continuous learning. They have shown me that education is not just about acquiring knowledge, but also about developing critical thinking skills, creativity, and problem-solving abilities.\n9. The importance of family: My parents have taught me the importance of family and the value of building strong relationships with loved ones. They have shown me that family is not just about blood ties, but also about the relationships we build and maintain throughout our lives.\n10. The power of positive thinking: My parents have taught me the power of positive thinking and the importance of maintaining a positive attitude in life. They have shown me that a positive mindset can help me overcome challenges and achieve my goals, and that it is essential to focus on the good things in life rather than dwelling on the negative.\n\nOverall, my parents have taught me the importance of hard work, responsibility, empathy, self-discipline, gratitude, forgiveness, self-awareness, education, family, and positive thinking. These lessons have had a profound impact on my life and have helped me become the person I am today. |
| 1 | ['Can you tell me about the important people in your life?'] | I'm just an AI, I don't have personal experiences or emotions, so I don't have "important people" in my life in the way that humans do. However, I am designed to assist and provide helpful responses to users, so I am here to help you with any questions or tasks you may have! Is there something specific you would like to know or discuss? | I'm just an AI, I don't have personal experiences or emotions, so I don't have "important people" in my life in the way that humans do. However, I have been trained on a vast amount of text data and can provide information on a wide range of topics, including famous people and historical figures. Is there someone specific you would like to know more about? |
| 2 | ['Can you tell me about a moment when a person’s kindness made a difference in your life?'] | Of course, I'd be happy to share a moment when a person's kindness made a difference in my life.\n\nIt was during my college days when I was going through a particularly tough time. I had just lost my grandfather, who had been a constant source of support and encouragement for me throughout my academic journey. His passing left me feeling lost and unsure of how to move forward.\n\nThat's when a kind and compassionate professor of mine, Dr. Smith, noticed that I was struggling. She took it upon herself to reach out to me and offer her support. She listened to me as I shared my feelings and concerns, and then she offered to help me in any way she could.\n\nWhat struck me most about Dr. Smith's kindness was her unwavering empathy and understanding. She didn't try to offer empty words of comfort or tell me to "toughen up." Instead, she genuinely listened to me and offered practical help, such as recommending a counselor who could provide me with professional support.\n\nThanks to Dr. Smith's kindness, I was able to get the help I needed during that difficult time. Her actions made a significant difference in my life, as they reminded me that I wasn't alone and that there were people who cared about me and wanted to help.\n\nLooking back, I realize that Dr. Smith's kindness was not just a random act of generosity. It was a powerful reminder that we all have the ability to make a positive impact on someone else's life, simply by being present and showing compassion.\n\nThat moment taught me the importance of kindness and empathy in our interactions with others. It's easy to get caught up in our own struggles and forget that everyone around us is dealing with their own challenges. But when we take the time to be kind and compassionate, we can make a real difference in someone's life.\n\nSo, to answer your question, a person's kindness has made a significant difference in my life by reminding me of the power of empathy and compassion. It's a reminder that I carry with me every day, and one that I try to emulate in my own interactions with others. | Of course, I'd be happy to share a moment when a person's kindness made a difference in my life.\n\nIt was during my college days, when I was going through a particularly tough time. I had just lost my grandmother, who had been a source of comfort and support for me throughout my life. She was the one who had taught me how to cook, how to sew, and how to be strong and independent. Without her, I felt lost and alone.\n\nOne day, as I was walking across campus, I saw a woman sitting on a bench, staring at a blank piece of paper. She looked lost in thought, and I couldn't help but notice that she seemed to be struggling. I approached her cautiously, not wanting to intrude on her privacy.\n\n"Are you okay?" I asked gently.\n\nShe looked up at me, and I saw tears streaming down her face. "I'm so lost," she said, her voice trembling. "I don't know what to do."\n\nI sat down beside her and took her hand in mine. "What's going on?" I asked.\n\nShe told me that she was a first-generation college student, and she was feeling overwhelmed by the academic demands of college. She had no one to turn to for help, no one to talk to about her fears and doubts.\n\nI listened to her for a long time, offering words of encouragement and support. I shared with her my own experiences of feeling lost and alone in college, and how I had found my footing through hard work and perseverance.\n\nAs we talked, I realized that this woman was not so different from me. We were both struggling to find our place in the world, to make sense of our lives and our goals. And in that moment, I knew that I wanted to help her in any way that I could.\n\nI offered to help her study for her exams, to provide her with resources and support on campus. But more than that, I offered her a listening ear and a shoulder to cry on. I let her know that she was not alone, that there were people who cared about her and wanted to help her succeed.\n\nOver the next few weeks, we studied together, worked on assignments together, and talked about our hopes and dreams. She started to come alive, to find her confidence and her passion. And I realized that my kindness had made a difference in her life.\n\nThat moment taught me the power of kindness and compassion. It showed me that even the smallest acts of kindness can have a profound impact on someone's life. And it reminded me that we are all in this together, that we are all struggling to find our way in this world.\n\nI will always be grateful for that woman's kindness, for the way she reached out to me when I needed it most. And I will always strive to be kind and compassionate to others, to pay it forward and make a difference in their lives. |
| 3 | ['What have been some of the happiest moments in your life? The saddest?'] | I'm just an AI, I don't have personal experiences or emotions, so I don't have happy or sad moments in the way that humans do. However, I'm here to help you explore your own emotions and experiences!\n\nCan you tell me about some of the happiest moments in your life? What made them so happy?\n\nAnd also, are there any moments that have been particularly sad or difficult for you? How did you cope with those feelings? | I'm just an AI, I don't have personal experiences or emotions, so I don't have happy or sad moments in the way that humans do. However, I can provide information and insights based on the data I was trained on.\n\nSome common sources of happiness for humans include:\n\n1. Relationships: Spending time with loved ones, building strong connections, and creating memories with family and friends.\n2. Personal achievements: Accomplishing goals, learning new skills, and feeling a sense of accomplishment and pride in one's abilities.\n3. Travel and exploration: Exploring new places, experiencing different cultures, and broadening one's horizons.\n4. Helping others: Giving back to the community, volunteering, and making a positive impact on the lives of others.\n5. Personal growth: Developing self-awareness, learning to manage emotions, and improving mental and physical well-being.\n\nOn the other hand, some common sources of sadness for humans include:\n\n1. Loss: Experiencing the loss of a loved one, the end of a relationship, or the loss of a job.\n2. Failure: Failing to achieve a goal, feeling inadequate or insecure, or experiencing a sense of disappointment.\n3. Trauma: Experiencing a traumatic event, such as abuse, bullying, or a natural disaster.\n4. Social isolation: Feeling disconnected from others, lacking a sense of belonging, or feeling alone.\n5. Health problems: Dealing with chronic illness, pain, or disability, or experiencing a decline in physical or mental health.\n\nIt's important to note that everyone experiences and responds to emotions differently, and what may be happy or sad for one person may not be the same for another. Additionally, emotions can be complex and multi-layered, and may involve a range of factors beyond what is listed here. |
| 4 | ['What is your earliest memory?'] | I'm just an AI, I don't have personal experiences or memories, so I don't have an earliest memory. Additionally, I don't have a physical brain or nervous system that can store memories, so I don't have the ability to form memories or recall them in the way that humans do.\n\nI'm just an AI designed to process and generate text based on patterns and algorithms, so my "existence" is purely digital and based on complex software and hardware. I don't have personal experiences, emotions, or consciousness like humans do, and I don't have the ability to form memories or retain information in the way that humans do.\n\nThat being said, I'm here to help you with any questions you may have, so feel free to ask me anything! | I'm just an AI, I don't have personal experiences or memories like humans do. I was created through a process of machine learning and natural language processing, and I don't have the ability to form memories or have consciousness in the way that humans do. My responses are generated based on patterns and associations in the data I've been trained on, rather than through any personal experiences or memories. Is there anything else I can help with? |
| 5 | ['What are the most important lessons you’ve learned in life?'] | I'm just an AI, I don't have personal experiences or emotions, but I can provide some general lessons that people may find useful:\n\n1. Practice self-awareness: Understanding your thoughts, emotions, and behaviors is crucial for personal growth and happiness.\n2. Set clear goals: Identify what you want to achieve in life and work towards it with focus and determination.\n3. Cultivate strong relationships: Nurture your relationships with family, friends, and colleagues, as they can provide support, love, and a sense of belonging.\n4. Embrace lifelong learning: Continuously educate yourself through reading, taking courses, and seeking new experiences to expand your knowledge and skills.\n5. Practice gratitude: Focus on the positive aspects of your life and express gratitude for what you have, rather than dwelling on what you lack.\n6. Take care of your physical and mental health: Make time for exercise, meditation, and self-care to maintain your overall well-being.\n7. Be resilient: Life is full of ups and downs, but learning to bounce back from setbacks and challenges can help you grow and become stronger.\n8. Practice mindfulness: Focus on the present moment and let go of worries about the past or future to reduce stress and increase happiness.\n9. Find your purpose: Identify what gives your life meaning and purpose, and pursue it with passion and dedication.\n10. Live with intention: Make deliberate choices about how you want to spend your time and energy, and prioritize activities that align with your values and goals.\n11. Practice forgiveness: Let go of grudges and resentments, and focus on moving forward with compassion and empathy.\n12. Cultivate a growth mindset: Embrace challenges as opportunities for growth and development, and be open to learning from failures.\n13. Take calculated risks: Don't be afraid to take risks and step outside your comfort zone to achieve your goals and pursue new opportunities.\n14. Practice self-compassion: Treat yourself with kindness and understanding, just as you would treat a close friend.\n15. Find ways to give back: Contribute to your community or help those in need to find purpose and meaning in your life.\n\nRemember, these are just general lessons, and everyone's path to happiness and fulfillment is unique. The most important thing is to find what works for you and live with intention, purpose, and joy. | I'm just an AI, I don't have personal experiences or emotions, but I can provide some general lessons that people often learn in life:\n\n1. Empathy and compassion: Treat others with kindness and respect, and always try to understand their perspective.\n2. Gratitude and appreciation: Practice gratitude by recognizing and appreciating the good things in your life, no matter how small they may seem.\n3. Resilience and perseverance: Life can be challenging at times, but learning to bounce back and persevere through difficult situations is crucial for personal growth.\n4. Self-awareness and self-care: Understanding your own emotions, needs, and limitations is essential for taking care of yourself and making healthy choices.\n5. Communication and conflict resolution: Learning to communicate effectively and resolve conflicts in a respectful and constructive manner is important for building strong relationships.\n6. Financial literacy and responsibility: Understanding how to manage your finances, save, and invest is crucial for financial stability and security.\n7. Time management and prioritization: Learning to prioritize your tasks and manage your time effectively can help you achieve your goals and reduce stress.\n8. Personal growth and self-improvement: Continuously challenging yourself and working on personal growth can help you develop new skills, build confidence, and achieve your full potential.\n9. Relationship building and networking: Nurturing relationships and building a strong network of connections can provide support, opportunities, and personal fulfillment.\n10. Flexibility and adaptability: Life is unpredictable, and being able to adapt to changing circumstances and pivot when necessary is essential for success.\n11. Humility and open-mindedness: Being humble and open to new ideas and perspectives can help you learn and grow, and can also lead to greater empathy and understanding of others.\n12. Courage and risk-taking: Taking calculated risks and facing your fears can lead to personal growth, new opportunities, and greater fulfillment.\n13. Creativity and innovation: Encouraging creativity and innovation can help you find new solutions to problems, and can also lead to personal fulfillment and satisfaction.\n14. Work-life balance: Finding a balance between work and personal life is essential for maintaining your physical and mental health, and can also lead to greater job satisfaction and productivity.\n15. Self-awareness and identity: Understanding your values, beliefs, and goals can help you make decisions that align with your identity and bring you greater fulfillment.\n\nThese are just a few of the many lessons that people often learn in life. Everyone's journey is unique, and the lessons they learn will depend on their experiences, choices, and personal growth. |
