llm install llm-gguf breaking llm

[lakamsani]

Hi, I tried the instructions from your email on July 23rd (earlier this week) titled Llama 3.1, now available in LLM

If you've [already installed LLM](https://substack.com/redirect/a4fc3808-3c94-4ba0-8828-e8a4557dc83d?j=eyJ1IjoiYmYzaSJ9.FZHXRpAef5uQ6flxkrGAyDGxn6x5piQQIjfeFfgCuKE) the following set of commands should get you setup with Llama 3.1 8B:

llm install llm-gguf

llm gguf download-model \
  https://huggingface.co/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --alias llama-3.1-8b-instruct --alias l31i

The first step appears to complete ok. But it broke the working llm installation. This is an arm54 M1 Pro MacBook with Sonoma 14.5

llm -c -m 4o
Traceback (most recent call last):
  File "/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/llama_cpp.py", line 75, in _load_shared_library
    return ctypes.CDLL(str(_lib_path), **cdll_args)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib, 0x0006): tried: '/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib' (no such file), '/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/bin/llm", line 5, in <module>
    from llm.cli import cli
  File "/opt/homebrew/lib/python3.12/site-packages/llm/__init__.py", line 18, in <module>
    from .plugins import pm
  File "/opt/homebrew/lib/python3.12/site-packages/llm/plugins.py", line 17, in <module>
    pm.load_setuptools_entrypoints("llm")
  File "/opt/homebrew/lib/python3.12/site-packages/pluggy/_manager.py", line 421, in load_setuptools_entrypoints
    plugin = ep.load()
             ^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llm_gguf.py", line 4, in <module>
    from llama_cpp import Llama
  File "/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/llama_cpp.py", line 88, in <module>
    _lib = _load_shared_library(_lib_base_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/llama_cpp.py", line 77, in _load_shared_library
    raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
RuntimeError: Failed to load shared library '/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib': dlopen(/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib, 0x0006): tried: '/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib' (no such file), '/Users/vamsee.lakamsani/Library/Python/3.12/lib/python/site-packages/llama_cpp/lib/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))