Choosing Your Inference Backend
Three backends dominate local inference in 2026: Ollama, llama.cpp with its server mode, and vLLM. Each has a distinct operational profile.
Ollama is the simplest to operate. It packages model management, inference, and an OpenAI-compatible REST API into a single binary with systemd integration. It uses llama.cpp under the hood but abstracts model pulling and versioning. The attack surface is larger than raw llama.cpp, but for most teams it is the right tradeoff.
llama.cpp in server mode gives you maximum control. You compile it yourself, link only the backends you need (CUDA, ROCm, Vulkan, CPU), and run a minimal HTTP server. No model registry, no automatic updates, no background daemons you did not start yourself. This is the correct choice for air-gapped environments.
vLLM is the production inference server for multi-GPU setups and high concurrency. If you are running AI inference as a shared service for a team, vLLM with tensor parallelism across two or four GPUs is the right architecture. It requires Python 3.11+ and a CUDA 12.x environment.
For a single-operator workstation or a small team on a shared server, Ollama is the fastest path to a working, isolated setup. For air-gapped production use, compile llama.cpp from source.
# Ollama installation without piping to bash
curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama.tgz
tar -C /usr/local/bin -xzf ollama.tgz
# Create a dedicated system user with no login shell
useradd -r -s /sbin/nologin -d /var/lib/ollama ollama
mkdir -p /var/lib/ollama
chown ollama:ollama /var/lib/ollamaNetwork Isolation: Bind to Localhost Only
The default Ollama configuration binds to 0.0.0.0:11434. On any multi-tenant server or box with external interfaces, this is wrong. Fix it before starting the service.
Create a systemd override rather than editing the unit file directly, so package updates do not revert your changes.
After applying the override, verify with ss before pulling any models. You want to see 127.0.0.1:11434 in the output, not 0.0.0.0:11434.
For llama.cpp server mode, the flag is --host 127.0.0.1. There is no environment variable path; you control it at launch time.
If you need to expose inference to other machines on a trusted LAN, tunnel it through SSH rather than binding to a network interface. This keeps the service localhost-only on the server and lets you use existing SSH key infrastructure for access control. Set up a SOCKS proxy or a direct port forward with ssh -L 11434:localhost:11434 user@ai-server rather than opening firewall rules.
# Create systemd override directory
mkdir -p /etc/systemd/system/ollama.service.d
# Write the override
cat > /etc/systemd/system/ollama.service.d/override.conf << 'EOF'
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/var/lib/ollama/models"
User=ollama
Group=ollama
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/ollama
EOF
systemctl daemon-reload
systemctl enable --now ollama
# Verify binding
ss -tlnp | grep 11434Systemd Hardening for the Inference Service
The systemd override above includes basic sandboxing flags. Extend them based on your threat model. The flags below are appropriate for a dedicated inference server where you control what models run.
ProtectKernelTunables and ProtectKernelModules prevent the process from modifying kernel parameters or loading modules. PrivateDevices removes access to raw device nodes. RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 blocks exotic socket families.
If you are running GPU inference with CUDA, PrivateDevices will block GPU access because CUDA needs /dev/nvidia*. In that case, drop PrivateDevices and add explicit device allow rules instead.
For CPU-only inference, the full hardening set applies cleanly. We tested this configuration on Debian 12 with llama.cpp 0.3.14 and Ollama 0.4.x running Llama 3.2 3B and Mistral 7B. The service starts cleanly with all restrictions in place.
[Service]
NoNewPrivileges=yes
PrivateTmp=yes
PrivateUsers=yes
ProtectHome=yes
ProtectSystem=strict
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=no
RestrictRealtime=yes
SystemCallFilter=@system-service
ReadWritePaths=/var/lib/ollama
CapabilityBoundingSet=
AmbientCapabilities=Audit Logging Every Inference Request
Knowing that your AI service is localhost-only is not enough. You need a log of what was queried and when, especially if multiple team members share access to the inference endpoint. This matters for compliance and for incident response if a compromised developer machine starts exfiltrating data through the AI API.
Ollama logs requests to its own log file, but auditd gives you a kernel-level record that cannot be tampered with by a compromised application. Add a rule to watch the Ollama socket and log file.
For teams running a shared inference server, consider placing nginx in front of Ollama as a logging proxy. This lets you capture full request/response pairs, enforce per-user API keys using nginx auth_request, and rate-limit queries. The nginx access log, combined with a structured log format, gives you a searchable audit trail that integrates with whatever log aggregation stack you already run.
We tested the nginx proxy pattern on our test server with three engineers sharing one inference endpoint. With the JSON log format below, we parsed query metadata into Loki and had a working dashboard in about 90 minutes.
# auditd rule: watch all network connections from the ollama process
# Add to /etc/audit/rules.d/ollama.rules
-a always,exit -F arch=b64 -S connect -F uid=ollama -k ollama_network
-w /var/log/ollama -p rwxa -k ollama_log
augenrules --load
# nginx proxy config snippet
server {
listen 127.0.0.1:11435;
access_log /var/log/nginx/ollama_access.log json_combined;
location / {
proxy_pass http://127.0.0.1:11434;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-User $http_x_user;
proxy_read_timeout 300s;
}
}Model Selection and Quantization for Operational Use
Not all models are equal for sysadmin tasks. For infrastructure work, command generation, and log analysis, 8B and 14B models in Q4_K_M or Q5_K_M quantization hit the right balance of speed and accuracy. Q4_K_M keeps most of the model quality while roughly halving VRAM requirements versus full precision.
On a 32GB RAM system with no GPU, Llama 3.1 8B in Q4_K_M fits entirely in RAM and runs at usable speeds. Mistral 7B Q5_K_M is slightly better at structured output and JSON generation, which matters for config generation tasks. For code and shell script generation, Qwen2.5-Coder 14B outperforms both on our test cases involving Ansible, Terraform, and complex bash.
Pull models explicitly and verify their SHA256 checksums against the published values on Hugging Face or the model provider's release page before using them. This is not optional. Poisoned models are a real supply chain threat.
Ollama stores models in /var/lib/ollama/models/blobs. Each blob is the raw GGUF file or a manifest layer. You can verify with sha256sum against the hash published in the model's manifest.
# Pull a specific model version
ollama pull llama3.1:8b-instruct-q4_K_M
# List models with their sizes
ollama list
# Find the blob path for verification
find /var/lib/ollama/models/blobs -name 'sha256-*' -size +1G
# Verify a specific model blob
sha256sum /var/lib/ollama/models/blobs/sha256-
# Compare output against the hash published in the model manifest
# For llama.cpp, pull the GGUF directly from Hugging Face
wget https://huggingface.co/bartowski/Llama-3.1-8B-Instruct-GGUF/resolve/main/Llama-3.1-8B-Instruct-Q4_K_M.gguf
sha256sum Llama-3.1-8B-Instruct-Q4_K_M.gguf Open WebUI: A Browser Interface That Stays Local
Open WebUI (formerly Ollama WebUI) is a self-hosted frontend that gives you a ChatGPT-like interface connected to your local inference backend. It runs as a Docker container or as a Python application, stores all history in a local SQLite database, and supports multi-user access with per-user API key management.
The critical configuration point is disabling telemetry and external connections at the application level, not just at the firewall. Open WebUI 0.5.x introduced an environment variable to disable all analytics calls.
For production team use, run Open WebUI behind an authenticated reverse proxy. Combine it with your existing SSO infrastructure using nginx and Vouch Proxy, or use Authelia if you already have that deployed. Do not expose Open WebUI directly to the internet, even with HTTPS. The session management in Open WebUI is adequate for internal use but has not been through the kind of penetration testing that justifies internet exposure.
We run Open WebUI on an internal VLAN accessible only via WireGuard. Engineers connect to the VPN, access the WebUI over HTTPS on port 443, and all inference stays within the private network.
# Docker deployment with telemetry disabled
docker run -d \
--name open-webui \
--restart unless-stopped \
-p 127.0.0.1:3000:8080 \
-e OLLAMA_BASE_URL=http://host-gateway:11434 \
-e WEBUI_SECRET_KEY=$(openssl rand -hex 32) \
-e ANONYMIZED_TELEMETRY=false \
-e SCARF_NO_ANALYTICS=true \
-e DO_NOT_TRACK=1 \
-v open-webui:/app/backend/data \
--add-host=host-gateway:host-gateway \
ghcr.io/open-webui/open-webui:0.5.4
# Confirm no external connections
ss -tnp | grep node
netstat -an | grep ESTABLISHED | grep -v 127.0.0.1AppArmor Profiles for Inference Processes
On Ubuntu and Debian systems, AppArmor provides mandatory access control that constrains what files, network sockets, and capabilities an inference process can access. A well-written AppArmor profile for Ollama limits it to exactly the paths it needs: its model directory, its socket, /tmp, and standard library paths.
The profile below denies network access except to localhost, denies writes everywhere except the designated model and log directories, and blocks execution of anything outside the standard binary paths. This stops a compromised or backdoored model from reaching out to the internet even if the systemd network restrictions somehow fail.
Load the profile in complain mode first, run your normal workload for a day or two, then review the audit log for denials. This catches edge cases like CUDA libraries that load from unexpected paths or temporary files written to non-standard locations.
Enforce mode after validating in complain mode. On our test server, the Ollama AppArmor profile in enforce mode with CPU inference had zero functional impact. With CUDA, we needed to add read access to /proc/driver/nvidia and /dev/nvidia* device nodes.
# /etc/apparmor.d/usr.local.bin.ollama
#include
/usr/local/bin/ollama {
#include
#include
network inet stream,
network inet6 stream,
network unix stream,
deny network inet dgram,
deny network inet6 dgram,
/var/lib/ollama/** rw,
/var/log/ollama/** rw,
/tmp/ollama* rw,
/usr/local/bin/ollama mr,
/lib/** mr,
/usr/lib/** mr,
/proc/self/** r,
/sys/devices/system/cpu/** r,
deny /etc/shadow r,
deny /root/** rw,
deny /home/** rw,
}
# Load and test
apparmor_parser -r /etc/apparmor.d/usr.local.bin.ollama
aa-status | grep ollama Integrating Local AI Into DevOps Workflows
The practical value of a local inference setup is in scripted integration, not just interactive chat. Sysadmins on our team use the Ollama API in shell scripts for log summarization, incident triage, and generating first-draft runbooks from raw change logs.
The Ollama API is OpenAI-compatible. Any tool that speaks to the OpenAI API can be pointed at your local endpoint by setting OPENAI_BASE_URL and OPENAI_API_KEY (any non-empty string works locally). This means tools built for cloud AI work with zero code changes against your local model.
For DevOps teams building AI-assisted automation pipelines, purpose-built platforms like taskbotshub.ai offer pre-built AI agent workflows for infrastructure tasks. If you need local inference integrated with CI/CD pipelines, ticketing systems, or on-call automation without building the integration layer yourself, that kind of tooling is worth evaluating before rolling your own.
For simple shell integration, the curl approach below works in any POSIX environment. We use a wrapper function in our team's shared bashrc that queries the local model and formats the output for terminal display. The function sends stdin to the model and writes the response to stdout, which means it composes naturally with Unix pipelines.
# Query local Ollama from a shell script
query_local_ai() {
local prompt="$1"
local model="${2:-llama3.1:8b-instruct-q4_K_M}"
curl -s http://127.0.0.1:11434/api/generate \
-H 'Content-Type: application/json' \
-d "$(jq -n --arg p "$prompt" --arg m "$model" \
'{model: $m, prompt: $p, stream: false}')" \
| jq -r '.response'
}
# Pipe a log file through the local model for summarization
tail -n 100 /var/log/syslog | query_local_ai \
"You are a Linux sysadmin. Summarize the key errors in this log output:"
# Use with OpenAI-compatible tools by overriding the endpoint
export OPENAI_BASE_URL=http://127.0.0.1:11434/v1
export OPENAI_API_KEY=local
# Now any OpenAI SDK call routes to your local modelAir-Gapped Deployment: No Internet Required
For environments where the server has no internet access, you need to pre-stage everything. This means downloading model files, container images, and binaries on an internet-connected machine and transferring them.
For llama.cpp, compile the binary on a machine with matching architecture and glibc version, or use a statically linked build. The llama.cpp release page on GitHub publishes pre-compiled binaries for common Linux targets. Download the GGUF model files separately and transfer them via your existing secure file transfer mechanism.
For Ollama in an air-gapped setup, export the Docker image on an internet-connected host, transfer the tar archive, and load it on the target. Model blobs transfer as plain files and live in /var/lib/ollama/models. Copy the entire directory tree rather than re-pulling.
When naming your internal AI service or project, treat it like any other infrastructure component. A clear, consistent naming convention matters for DNS, service discovery, and documentation. If you are also standing up any public-facing tooling around your AI infrastructure, a service like nicename.me can help identify available domain names that match your project naming conventions before you commit to a name internally.
Verify GPG signatures on all downloaded binaries before transfer. The llama.cpp releases are signed with the project's GPG key. Ollama's releases include SHA256 checksums in the release notes.
# On internet-connected machine: stage everything
# 1. Download Ollama binary
curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama-linux-amd64.tgz
sha256sum ollama-linux-amd64.tgz # verify against release page
# 2. Download model GGUF directly
wget 'https://huggingface.co/bartowski/Llama-3.1-8B-Instruct-GGUF/resolve/main/Llama-3.1-8B-Instruct-Q4_K_M.gguf'
# 3. Transfer to air-gapped host
rsync -avz --progress ollama-linux-amd64.tgz user@airgapped-host:/tmp/
rsync -avz --progress Llama-3.1-8B-Instruct-Q4_K_M.gguf user@airgapped-host:/var/lib/ollama/models/
# On air-gapped host: run llama.cpp server directly against the GGUF
./llama-server \
--host 127.0.0.1 \
--port 8080 \
--model /var/lib/ollama/models/Llama-3.1-8B-Instruct-Q4_K_M.gguf \
--n-gpu-layers 0 \
--threads $(nproc) \
--ctx-size 4096Monitoring and Resource Limits
Inference workloads can saturate CPU, RAM, and if you have a GPU, VRAM. Without resource limits, a runaway inference process or a script that accidentally spawns parallel requests can bring down a shared server.
Set memory limits in the systemd unit. For an 8B Q4_K_M model on CPU, set MemoryMax to 8GB. For a 14B Q5_K_M, use 12GB. These numbers give the model room to load with overhead.
For GPU inference, nvidia-smi dmon gives you per-second utilization metrics. Pipe it to a log file or integrate with your existing monitoring. If you run Prometheus, the dcgm-exporter container from NVIDIA gives you full GPU metrics in the standard format.
Set CPUQuota in the systemd unit to prevent the inference process from monopolizing all cores on a shared machine. 400% on an 8-core box (50% of total CPU) is a reasonable starting point for background inference. For interactive use by a single user, remove the limit.
For the Ollama API, set OLLAMA_MAX_QUEUE=4 and OLLAMA_NUM_PARALLEL=1 unless you have enough VRAM to run multiple contexts simultaneously. Queuing requests rather than spawning parallel inference processes keeps memory usage predictable.
# Add to systemd override.conf
[Service]
MemoryMax=8G
MemorySwapMax=0
CPUQuota=400%
TasksMax=64
# Environment variables for Ollama resource control
Environment="OLLAMA_MAX_QUEUE=4"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
# Monitor inference resource use in real time
watch -n 2 'ps aux | grep ollama | grep -v grep | awk "{print \$6/1024 \" MB RAM\", \$3 \"% CPU\"}"'
# GPU monitoring if applicable
nvidia-smi dmon -s pucvmet -d 5