[enhancement]: Script to help in compiling bitsandbytes for ROCm for InvokeAI

[webmaster-exit-1]

Is there an existing issue for this?

• [x] I have searched the existing issues

Contact Details

[email protected]

What should this feature add?

Fix: General bitsandbytes installation script for ROCm users

This PR introduces a script to resolve common bitsandbytes installation issues on ROCm-based systems. It addresses the externally-managed-environment error, AttributeError: 'bitsandbytes' has no attribute 'nn' from version conflicts, and RuntimeError: Configured ROCm binary not found by automating the build process from a known-compatible commit and performing the necessary file renaming.

The script automatically detects the virtual environment path and the user's ROCm architecture (gfx...), making it a general solution for various AMD GPUs.

Steps Taken:

A clean rebuild of bitsandbytes is performed by uninstalling the existing package and removing build artifacts.

A specific bitsandbytes commit from the rocm branch is checked out to ensure compatibility with InvokeAI's code.

The library is built with cmake and the user's specific ROCm architecture.

The built library file is renamed from libbitsandbytes_rocm64.so to libbitsandbytes_rocm63.so as a necessary workaround for current InvokeAI code.

The script concludes with a verification step to confirm a successful installation.

This fix should significantly reduce setup friction for community members using AMD hardware.

Alternatives

No response

Additional Content

#!/usr/bin/env bash

# This script performs a clean rebuild of the bitsandbytes library from a known-compatible
# source commit, specifically for ROCm-enabled systems with InvokeAI.
#
# It is designed to fix the following issues:
# 1. 'externally-managed-environment' errors during pip installation.
# 2. 'AttributeError: module 'bitsandbytes' has no attribute 'nn'' due to version mismatches.
# 3. 'RuntimeError: Configured ROCm binary not found' due to incorrect library filenames.

# Exit immediately if a command exits with a non-zero status.
set -e

# --- Step 0: Initial checks and variable setup ---

echo "Starting bitsandbytes build and fix script for InvokeAI on ROCm."

# Automatically detect the virtual environment path
if [[ -z "${VIRTUAL_ENV}" ]]; then
    echo "Warning: No Python virtual environment is active. Attempting to locate the venv."
    VENV_PATH=$(dirname $(dirname $(command -v invokeai-web)))
    if [[ -z "${VENV_PATH}" || ! -d "${VENV_PATH}" ]]; then
        echo "Error: Could not locate InvokeAI virtual environment."
        echo "Please activate your venv before running this script (e.g., 'source .venv/bin/activate')."
        exit 1
    fi
else
    VENV_PATH="${VIRTUAL_ENV}"
fi

PYTHON_BIN="${VENV_PATH}/bin/python3"
SITE_PACKAGES="${VENV_PATH}/lib/python3.12/site-packages"

# Automatically detect ROCm architecture (gfx model)
ROCM_ARCH=$(rocminfo | grep 'gfx' | tail -n 1 | awk '{print $1}' | tr -d '[:space:]')
if [[ -z "${ROCM_ARCH}" ]]; then
    echo "Error: Could not automatically detect your ROCm architecture."
    echo "Please run 'rocminfo | grep gfx' and manually set the ROCM_ARCH variable in the script."
    exit 1
fi

echo "Detected InvokeAI venv path: ${VENV_PATH}"
echo "Detected ROCm architecture: ${ROCM_ARCH}"

# --- Step 1: Clean Up Existing Installation ---

echo "Removing existing bitsandbytes packages and build artifacts..."

# Uninstall bitsandbytes via pip using the full path to the interpreter
"${PYTHON_BIN}" -m pip uninstall -y bitsandbytes || echo "bitsandbytes not found, continuing..."

# Remove the bitsandbytes directory to ensure a clean slate
RM_PATH="${SITE_PACKAGES}/bitsandbytes/"
if [ -d "$RM_PATH" ]; then
    echo "Removing old bitsandbytes installation from ${RM_PATH}"
    rm -rf "$RM_PATH"
else
    echo "Directory not found, proceeding with clean clone."
fi

# Remove any old clone if it exists
if [ -d "bitsandbytes" ]; then
    rm -rf bitsandbytes
fi

# --- Step 2: Clone and Build from Source ---

echo "Cloning bitsandbytes repository..."
git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd bitsandbytes/

# Check out a specific, known-compatible commit from the ROCm branch.
# This commit is known to work with InvokeAI and contains the necessary 'nn' submodule.
echo "Checking out known-compatible commit..."
git checkout a9e10398f6d76ae58066598c114f0896010d8624

echo "Installing build dependencies..."
"${PYTHON_BIN}" -m pip install -r requirements-dev.txt || echo "Could not install dependencies. Proceeding with next step."

echo "Building bitsandbytes for ROCm with arch=${ROCM_ARCH}..."
cmake -DCOMPUTE_BACKEND=hip -S . -DBNB_ROCM_ARCH="${ROCM_ARCH}" -DCMAKE_INSTALL_PREFIX="${VENV_PATH}" -DCMAKE_INSTALL_LIBDIR="lib/python3.12/site-packages/bitsandbytes/"
make -j$(nproc)

# --- Step 3: Rename the built library ---
# The build output generates libbitsandbytes_rocm64.so, but the python code within
# InvokeAI's current version expects libbitsandbytes_rocm63.so.
echo "Renaming built library to expected name..."
mv "${VENV_PATH}/bin/bitsandbytes/bitsandbytes/libbitsandbytes_rocm64.so" "${VENV_PATH}/bin/bitsandbytes/bitsandbytes/libbitsandbytes_rocm63.so"

echo "Installation complete. Running a quick check."
# Added --break-system-packages to override the externally-managed-environment error
"${PYTHON_BIN}" -m pip install -e . --break-system-packages

# --- Step 4: Verification ---
echo "Verifying bitsandbytes installation..."
"${PYTHON_BIN}" -c "import bitsandbytes as bnb; print(f'bitsandbytes version: {bnb.__version__}'); print(f'bitsandbytes library path: {bnb.__file__}');"
echo "If no errors were reported, the installation was successful."

[webmaster-exit-1]

This helped me. I hope it helps others.

[webmaster-exit-1]

Thought I'd mention this:

Once the script is finished this is what you should see at the end if everything worked. (I omitted my person path with YOUR_PATH_TO)

Verifying bitsandbytes installation...
bitsandbytes version: 0.48.0.dev0
bitsandbytes library path: /YOUR_PATH_TO/Invokeai/.venv/bin/bitsandbytes/bitsandbytes/__init__.py
If no errors were reported, the installation was successful.

Startup invokeai-web --root /path/to/invokeai --config /path/to/invokeai/invokeai.yaml And you should NOT see any bitsandbytes errors OR warnings.

Happy Hacking.

[slokhorst]

Thanks for this, however the commit hash seems to be incorrect:

Checking out known-compatible commit...
fatal: unable to read tree (a9e10398f6d76ae58066598c114f0896010d8624)

The commit does not exist: https://github.com/bitsandbytes-foundation/bitsandbytes/commit/a9e10398f6d76ae58066598c114f0896010d8624

Typo maybe?

[webmaster-exit-1]

Hey, ya I noticed that too, the repo/commit used in my script had changed. Just use the default bitsandbytes repo instead of the commit. Edit the git checkout to git clone and cd into it. The rest of the script "should" work then. Depending on your specific needs, the script may need tweaking. Keep that in mind.

The main thing one needs to do is compile bitsandbytes for ROCm.

echo "Building bitsandbytes for ROCm with arch=${ROCM_ARCH}..." 

cmake -DCOMPUTE_BACKEND=hip -S . -DBNB_ROCM_ARCH="${ROCM_ARCH}" -DCMAKE_INSTALL_PREFIX="${VENV_PATH}" -DCMAKE_INSTALL_LIBDIR="lib/python3.12/site-packages/bitsandbytes/" 

make -j$(nproc)

Then move and/or rename the compiled library file *.so to the correct path. At the time of writing this InvokeAI is using ROCm version 6.3 if the compile .so is rocm6.4 you can create a symbolic link to the path ln -s /path/to/bitsandbytes_rocm64.so /path/to/bitsandbytes_rocm63.so

If you don't want to use the latest ROCm 6.4 (defaults since it is the latest version) and use what invokeai is using (ROCm 6.3) add the download url in the pip commands at the end. --index-url https://download.pytorch.org/whl/rocm6.3 and change the renaming part in the script.

Or whatever works best for you.

• Oh, and 8bit quants still won't work. Forgot to mention that.

• If everything completely fails; throw this entire page into AI and have it sort it out. It might or might not work. Still worth a shot, maybe.

(I'm writing this from memory and not at my pc)

Thanks for this, however the commit hash seems to be incorrect:

Checking out known-compatible commit...
fatal: unable to read tree (a9e10398f6d76ae58066598c114f0896010d8624)

The commit does not exist: https://github.com/bitsandbytes-foundation/bitsandbytes/commit/a9e10398f6d76ae58066598c114f0896010d8624

Typo maybe?

[webmaster-exit-1]

Something else to note, I believe there is a precompiled python-bitsandbytes-rocm-git package in the Arch Linux AUR. So that may be another avenue to checkout. Hope all this info helps.

[max-maag]

Your script didn't run on my machine so I made some changes:

• Added set -o pipefail to also fail on error in pipe.
• Improved parsing of ROCm architecture.
• Removed manual deletion of bitsandbytes directory as pip uninstall already takes care of that.
• Added a line to ensure pip is installed.
• Added arguments to git clone to only download the latest commit but not the entire history.
• Removed manual installation of bitsandbytes dependencies via requirements-dev.txt as that file does not exist anymore.
• Added an optional argument to the script so you can specify a build directory for bitsandbytes.
• Removed irrelevant CMAKE_INSTALL_* arguments.
• Replaced manually copying the .so file by a call to pip install and subsequent creation of a symlink.
• Replaced verification by an invocation of the bitsandbytes model so more debugging information is printed.
• Removed the --break-system-packages argument to pip install because that does not seem like the correct solution to pip complaining about an externally managed env. The local virtual env should not be externally managed.

This seems to install bitsandbytes correctly as the module is loaded without issues. However, invoking unfortunately still fails because of a different issue so I am not 100% sure the script is actually correct.

Anyways, here it is, maybe it works for you:

#!/usr/bin/env bash

# This script performs a clean rebuild of the bitsandbytes library.
#
# It is designed to fix the following issues:
# 1. 'externally-managed-environment' errors during pip installation.
# 2. 'AttributeError: module 'bitsandbytes' has no attribute 'nn'' due to version mismatches.
# 3. 'RuntimeError: Configured ROCm binary not found' due to incorrect library filenames.

# Exit immediately if a command exits with a non-zero status.
set -eo pipefail

# --- Step 0: Initial checks and variable setup ---

echo "Starting bitsandbytes build and fix script for InvokeAI on ROCm."

# Automatically detect the virtual environment path
if [[ -z "${VIRTUAL_ENV}" ]]; then
    echo "Warning: No Python virtual environment is active. Attempting to locate the venv."
    VENV_PATH=$(dirname $(dirname $(command -v invokeai-web)))
    if [[ -z "${VENV_PATH}" || ! -d "${VENV_PATH}" ]]; then
        echo "Error: Could not locate InvokeAI virtual environment."
        echo "Please activate your venv before running this script (e.g., 'source .venv/bin/activate')."
        exit 1
    fi
else
    VENV_PATH="${VIRTUAL_ENV}"
fi

PYTHON_BIN="${VENV_PATH}/bin/python3"
SITE_PACKAGES="${VENV_PATH}/lib/python3.12/site-packages"

# Automatically detect ROCm architecture (gfx model)
ROCM_ARCH=$(rocminfo | grep -Po '(?<!-)gfx[0-9a-z]+' | tail -n 1)
if [[ -z "${ROCM_ARCH}" ]]; then
    echo "Error: Could not automatically detect your ROCm architecture."
    echo "Please run 'rocminfo | grep gfx' and manually set the ROCM_ARCH variable in the script."
    exit 1
fi

echo "Detected InvokeAI venv path: '${VENV_PATH}'"
echo "Detected ROCm architecture: '${ROCM_ARCH}'"

# Remove any old clone if it exists
if [ -d "bitsandbytes" ]; then
    rm -rf bitsandbytes
fi

# --- Step 2: Clone and Build from Source ---

echo "Installing build dependencies..."
"$PYTHON_BIN" -m ensurepip

build_dir=${1:-$(realpath bitsandbytes)}

echo "Cloning bitsandbytes repository into '$build_dir'..."
git clone --depth 1 https://github.com/bitsandbytes-foundation/bitsandbytes.git "$build_dir"

pushd "$build_dir"

echo "Building bitsandbytes for ROCm with arch=${ROCM_ARCH}..."
cmake -DCOMPUTE_BACKEND=hip -S . -DBNB_ROCM_ARCH="${ROCM_ARCH}"
make -j$(nproc)

popd

# --- Step 3: Install to virtual env

echo "Installing bitsandbytes to virtual env..."
"${PYTHON_BIN}" -m pip install --no-deps --force-reinstall "$build_dir"

ln -sf "libbitsandbytes_rocm64.so" "$SITE_PACKAGES/bitsandbytes/libbitsandbytes_rocm63.so"

# --- Step 4: Verification ---
echo "Verifying bitsandbytes installation..."
"${PYTHON_BIN}" -m bitsandbytes
echo "If no errors were reported, the installation was successful."

[webmaster-exit-1]

Cool man. However it works. Glad to see it helped. Too bad the compiled bitsandbytes still won't work (I think) with Quant 8 or FP 8. But remember the main part. COMPILING BITSANDBYTES FOR ROCM. The rest of the script is just fancy automation and unnecessary and was done just to look good, and meant for my messy filesystem. So like you said there were unneeded redundant flags, ect... Happy you caught that. I was too busy to edit it into a perfect script. MAIN THING YOU ONLY NEED TO DO --> Compile, then just move the libbitsandbytes_rocm63.so to where the error complains about.