The Best Way to Get Deepseek-R1: By Using an LXC in Proxmox

[Josef_Founder]

A script I made to automate GPU Passthrough:

UPDATE: JUST FOLLOW THIS LINK, PLEASE

sudo nano gpu.sh

Paste the below script (gpu.sh) into our nano file

Give gpu.sh permission to do its thing:

chmod +x gpu.sh

Replace LXC_ID with your Container Number (e.g. Proxmox defaults to 100)

sudo bash gpu.sh <LXC_ID>

gpu.sh

#!/bin/bash

set -e

LXC_ID=$1  # LXC container ID passed as an argument
if [ -z "$LXC_ID" ]; then
    echo "Usage: $0 <LXC_ID>"
    exit 1
fi

# Enable IOMMU
if grep -q "GenuineIntel" /proc/cpuinfo; then
    echo "Detected Intel CPU. Enabling Intel IOMMU."
    IOMMU_PARAM="intel_iommu=on iommu=pt"
elif grep -q "AuthenticAMD" /proc/cpuinfo; then
    echo "Detected AMD CPU. Enabling AMD IOMMU."
    IOMMU_PARAM="amd_iommu=on iommu=pt"
else
    echo "Unknown CPU vendor. Exiting."
    exit 1
fi

sed -i "s/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet $IOMMU_PARAM\"/" /etc/default/grub
proxmox-boot-tool refresh
echo "IOMMU enabled. Reboot required."

# Detect GPU
GPU_TYPE="unknown"
if lspci | grep -i nvidia; then
    GPU_TYPE="nvidia"
elif lspci | grep -i amd; then
    GPU_TYPE="amd"
elif lspci | grep -i 'Intel Corporation' | grep -i 'VGA'; then
    GPU_TYPE="intel"
fi

echo "Detected GPU: $GPU_TYPE"

# Configure LXC container for GPU passthrough
LXC_CONF="/etc/pve/lxc/$LXC_ID.conf"

if [ ! -f "$LXC_CONF" ]; then
    echo "LXC container config not found: $LXC_CONF"
    exit 1
fi

echo "Configuring LXC GPU passthrough..."
cat <<EOL >> $LXC_CONF
lxc.cgroup2.devices.allow = c 226:* rwm
lxc.mount.entry = /dev/dri dev/dri none bind,optional,create=dir
EOL

if [ "$GPU_TYPE" == "nvidia" ]; then
    cat <<EOL >> $LXC_CONF
lxc.cgroup2.devices.allow = c 195:* rwm
lxc.mount.entry = /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry = /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
EOL
fi

# Restart LXC container
pct stop $LXC_ID
pct start $LXC_ID

# Install drivers inside LXC
echo "Installing GPU drivers in LXC container..."
if [ "$GPU_TYPE" == "nvidia" ]; then
    pct exec $LXC_ID -- bash -c "apt update && apt install -y nvidia-driver nvidia-cuda-toolkit"
elif [ "$GPU_TYPE" == "amd" ]; then
    pct exec $LXC_ID -- bash -c "
        apt update && apt install -y wget && \
        wget https://repo.radeon.com/amdgpu-install/6.0/ubuntu/jammy/amdgpu-install_6.0.60000-1_all.deb && \
        dpkg -i amdgpu-install_6.0.60000-1_all.deb && \
        amdgpu-install --usecase=rocm"
elif [ "$GPU_TYPE" == "intel" ]; then
    pct exec $LXC_ID -- bash -c "
        apt update && apt install -y intel-media-va-driver && \
        echo 'Intel GPUs require building Ollama from source with intel-basekit. Use the script from: https://github.com/tteck/Proxmox/blob/main/install/ollama-install.sh'"
fi

# Verify GPU availability inside LXC
if [ "$GPU_TYPE" == "nvidia" ]; then
    pct exec $LXC_ID -- nvidia-smi
elif [ "$GPU_TYPE" == "amd" ]; then
    pct exec $LXC_ID -- clinfo
elif [ "$GPU_TYPE" == "intel" ]; then
    pct exec $LXC_ID -- vainfo
fi

echo "GPU passthrough setup complete for LXC ID $LXC_ID. A reboot is recommended."

Now we are ready to set up Open WebUI, itself:

To enable your graphics card(s) in Open WebUI, you need to configure the Ollama backend (if using Ollama for LLM inference) or CUDA/ROCM for models that support GPU acceleration.

1. Verify GPU Availability

Before configuring Open WebUI, ensure your GPU is accessible inside the LXC container:

For NVIDIA GPUs:

Run:

bash

nvidia-smi

If your GPU appears in the output, it’s recognized.

For AMD GPUs (ROCM-based):

Run:

bash

clinfo

or

bash

rocminfo

For Intel GPUs:

Run:

bash

vainfo

---

2. Configure Ollama for GPU Acceleration

If you’re using Ollama in Open WebUI, you need to ensure GPU acceleration is enabled.

For NVIDIA GPUs (CUDA)

Enable CUDA support in Ollama:

bash

OLLAMA_CUDA=1 ollama run llama3

Or set it permanently:

bash

echo "export OLLAMA_CUDA=1" >> ~/.bashrc
source ~/.bashrc

For AMD GPUs (ROCM)

Enable ROCm support:

bash

OLLAMA_ROCM=1 ollama run llama3

Or set it permanently:

bash

echo "export OLLAMA_ROCM=1" >> ~/.bashrc
source ~/.bashrc

---

3. Update Open WebUI Settings

1. Open Open WebUI in your browser.
2. Go to Settings (gear icon in the UI).
3. Look for Backend Settings.
4. Ensure that Ollama (or the relevant LLM engine) is selected.
5. If there’s an option for GPU Acceleration, enable it.

---

4. Restart Open WebUI & Test GPU Usage

Restart Open WebUI to apply changes:

bash

systemctl restart open-webui

Then, check if the GPU is being utilized:

bash

nvidia-smi  # For NVIDIA
clinfo       # For AMD
vainfo       # For Intel

Alright, OK, I’ll just automate this for you…

run the following inside of the LXC container:

sudo nano gpuLXC.sh

chmod +x gpuLXC.sh

sudo ./gpuLXC.sh

gpuLXC.sh:

#!/bin/bash

set -e

# Ensure running as root
if [ "$EUID" -ne 0 ]; then
    echo "Please run as root or with sudo"
    exit 1
fi

# Update package lists
apt update

# Install necessary packages including lspci
apt install -y git python3 python3-pip wget clinfo vainfo pciutils

# Clone or update Open WebUI repository
if [ -d "open-webui" ]; then
    echo "Updating existing open-webui repository..."
    cd open-webui
    git fetch --all
    git reset --hard origin/main
    git pull
else
    echo "Cloning open-webui repository..."
    git clone --depth 1 --branch main https://github.com/open-webui/open-webui.git
    cd open-webui
fi

# Install Python dependencies
if [ -f "requirements.txt" ]; then
    pip3 install -r requirements.txt
else
    echo "No requirements.txt found. Installing common dependencies..."
    pip3 install fastapi uvicorn python-multipart jinja2 aiofiles
    echo "Please check if additional dependencies are needed for your specific setup."
fi

# Detect GPUs and install drivers
GPU_TYPES=()
if lspci | grep -i nvidia; then
    GPU_TYPES+=("nvidia")
    apt install -y nvidia-driver nvidia-cuda-toolkit
    pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
fi

if lspci | grep -i amd; then
    GPU_TYPES+=("amd")
    echo "Attempting to install AMD GPU drivers..."

    # Temporarily suppress errors from dkms and initramfs-tools
    export DEBIAN_FRONTEND=noninteractive
    apt -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install -y amdgpu-dkms amdgpu || {
        echo "Failed to install amdgpu-dkms with standard settings. Trying to resolve initramfs update issues..."
        
        # Force update of initramfs with different compression
        update-initramfs -u -k all -c -z lz4 || {
            echo "Failed to update initramfs with lz4 compression. Attempting with gzip..."
            update-initramfs -u -k all -c -z gzip || {
                echo "Failed to update initramfs. Manual intervention might be needed."
            }
        }

        # Try installing again
        apt -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install -y amdgpu-dkms amdgpu || {
            echo "Installation of AMD GPU drivers failed. Please check logs for manual fixes."
        }
    }
    pip3 install torch torchvision torchaudio rocm
fi

if lspci | grep -i 'Intel Corporation' | grep -i 'VGA'; then
    GPU_TYPES+=("intel")
    apt install -y intel-media-va-driver
    pip3 install openvino
fi

# Configure GPU environment for Python
cat <<EOL > gpu_config.py
import os

# Set CUDA_VISIBLE_DEVICES if NVIDIA GPU is present
if 'nvidia' in os.environ.get('GPU_TYPES', '').split(','):
    os.environ['CUDA_VISIBLE_DEVICES'] = 'all'

# For AMD and Intel, we'll use specific environment variables
os.environ['GPU_TYPES'] = '${GPU_TYPES[*]}'

# Additional setup for AMD or Intel might be required here

if 'amd' in os.environ.get('GPU_TYPES', '').split(','):
    os.environ['ROCM_PATH'] = '/opt/rocm'
    
if 'intel' in os.environ.get('GPU_TYPES', '').split(','):
    os.environ['INTEL_OPENVINO_DIR'] = '/opt/intel/openvino'

EOL

# Adjust Open WebUI's main script or config to include GPU settings
# Here's an example; adapt as necessary based on Open WebUI's actual structure
if [ -f "run.py" ]; then
    sed -i '/import os/a import gpu_config' run.py
else
    echo "Could not find 'run.py'. Please manually configure GPU usage in the appropriate script or config file."
fi

# Run Open WebUI (this is a placeholder; replace with actual run command)
if [ -f "run.py" ]; then
    echo "Starting Open WebUI with GPU support..."
    python3 run.py  # Note: This is a placeholder. Use the actual command to start Open WebUI
else
    echo "No run.py found. Please use the correct command to start Open WebUI."
fi

# Verify GPU usage
echo "Verifying GPU usage:"
for gpu_type in "${GPU_TYPES[@]}"; do
    case $gpu_type in
        "nvidia")
            nvidia-smi || echo "NVIDIA GPU detected but 'nvidia-smi' failed to run."
            ;;
        "amd")
            clinfo || echo "AMD GPU detected but 'clinfo' failed to run."
            ;;
        "intel")
            vainfo || echo "Intel GPU detected but 'vainfo' failed to run."
            ;;
    esac
done

echo "Open WebUI setup with GPU support is complete. Check the application interface for model performance."

Running Ollama and Open WebUI Self-Hosted With AMD GPU

Turn your computer into an AI machine - Configure and basic usage of Open WebUI