Extra Environment variables needed for kubernetes GPU

[tgm4883]

I needed some extra environment variables in order for KoboldAI to see my GPU (nvidia) inside the container. Figured this should be documented somewhere.

Specifically, I needed these

NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_VISIBLE_DEVICES=all

For reference, kubernetes manifest. I use metallb and some persistent storage, so the Deployment section may be the only part that anyone cares about. I'm running this on microk8s with the gpu plugin enabled.

---
apiVersion: v1
kind: Service
metadata:
  name: koboldai
  labels:
    app: koboldai
spec:
  type: LoadBalancer
  ports:
  - name: koboldai
    port: 5000
    protocol: TCP
  selector:
    app: koboldai
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: koboldai-content
  annotations:
    volume.beta.kubernetes.io/storage-class: "freenas-nfs-csi"
spec:
  storageClassName: freenas-nfs-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: koboldai
  labels:
    app: koboldai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: koboldai
  template:
    metadata:
      labels:
        app: koboldai
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nvidia.com/gpu.present
                operator: In
                values:
                - "true"
      containers:
      - name: koboldai
        image: harbor.local.mashos.com/koboldai/koboldai:united
        env:
        - name: KOBOLDAI_DATADIR
          value: "/content"
        - name: KOBOLDAI_MODELDIR
          value: "/content"
        - name: KOBOLDAI_ARGS
          value: "--quiet --host --override_delete --override_rename"
        - name: NVIDIA_DRIVER_CAPABILITIES
          value: "compute,utility"
        - name: NVIDIA_VISIBLE_DEVICES
          value: all
        ports:
        - containerPort: 5000
        volumeMounts:
          - name: koboldai-content
            mountPath: /content
      volumes:
        - name: koboldai-content
          persistentVolumeClaim:
            claimName: koboldai-content