System Requirements and FreeBSD Version

FreeBSD 14.2-RELEASE is the minimum we recommend. Older branches lack the linprocfs improvements that some Ollama builds depend on for Linux compatibility shims. You need at minimum 16GB RAM to run 7B parameter models at reasonable speed using CPU inference. For 13B models, 32GB is the practical floor. GPU acceleration requires an NVIDIA card from the RTX 3000 series or newer - AMD GPU compute on FreeBSD is still not production-ready as of mid-2026.

Verify your FreeBSD version before starting:

If you are on 14.1 or earlier, upgrade first. The freebsd-update path from 14.1 to 14.2 is clean and takes about 20 minutes on a standard server.

freebsd-version -k
uname -r
# Should output: 14.2-RELEASE

Bootstrapping pkg and Updating Ports

A fresh FreeBSD install ships without pkg. Bootstrap it first, then sync the ports tree. We use pkg for binary packages where available and ports for anything that needs custom compile flags, specifically for NVIDIA driver versions.

After bootstrapping, update your repository catalog and install essential build tools in one pass. The devel/git and ports-mgmt/portmaster packages will be needed later for ports-based builds.

pkg bootstrap
pkg update
pkg upgrade -y
pkg install -y git portmaster wget curl bash

NVIDIA Driver Installation on FreeBSD

FreeBSD ships nvidia-driver in the ports tree under x11/nvidia-driver. As of August 2026, the current version in ports is 550.107.02 for the main branch. Install the package version unless you need a specific CUDA-adjacent compute version, in which case build from ports with custom options.

After installing the driver, load the kernel module and add it to loader.conf so it persists across reboots. Then verify the card is recognized with nvidia-smi.

If nvidia-smi returns 'No devices were found', check that your card is not in a PCI slot blocked by IOMMU grouping. On AMD platforms, check the BIOS for SVM mode. On Intel, verify VT-d settings. We ran into this on our test server with an ASUS ProArt board - toggling 'Above 4G Decoding' in UEFI fixed the detection issue immediately.

pkg install -y nvidia-driver nvidia-settings
kldload nvidia
kldload nvidia-modeset
echo 'nvidia_load="YES"' >> /boot/loader.conf
echo 'nvidia-modeset_load="YES"' >> /boot/loader.conf
nvidia-smi
// advertisement

Installing Ollama on FreeBSD

Ollama reached ports under sysutils/ollama in late 2025. The 0.6.x branch added native FreeBSD support without requiring Linux compatibility layer. Install it via pkg:

Ollama runs as a server process. The default socket listens on 127.0.0.1:11434. The rc script installed by the package handles startup. Enable it in rc.conf and start it manually for the first time to watch the logs and confirm GPU detection.

Check the startup output carefully. You want to see lines containing 'CUDA device' or 'NVIDIA GPU detected'. If you only see 'CPU backend selected', Ollama did not find the driver - most often this means the nvidia kernel module was not loaded before the service started.

pkg install -y ollama
sysrc ollama_enable="YES"
service ollama start
service ollama status
# Watch logs:
tail -f /var/log/ollama.log

Pulling and Running Your First Model

Ollama uses a pull-then-run model similar to Docker. Models download from registry.ollama.ai and are stored under /var/db/ollama/models by default on FreeBSD. Make sure that directory has at least 50GB free before pulling anything larger than a 7B model.

Start with llama3.2:3b for a fast sanity check - it is 2.0GB and runs inference at roughly 45 tokens per second on a 4070 in our testing. Once that works, move to larger models based on your available VRAM. The 4070 has 12GB VRAM, which fits mistral:7b (4.1GB) with room for context.

For automated or scripted usage, the REST API is more useful than the interactive shell. Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions, which means any tool built for the OpenAI API works against a local Ollama server without code changes.

ollama pull llama3.2:3b
ollama run llama3.2:3b "What is the capital of France?"

# Pull a larger model after confirming the above works:
ollama pull mistral:7b

# API test:
curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "llama3.2:3b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Isolating Ollama Inside a Bastille Jail

Running Ollama directly on the host works but gives you no isolation. For production use, confine it in a thick jail using bastille. GPU passthrough into a jail on FreeBSD requires passing the /dev/nvidiactl and /dev/nvidia0 device nodes into the jail, plus setting the allow.mlock and sysvshm jail parameters.

Install bastille and bootstrap a base release first. We use FreeBSD 14.2-RELEASE as the jail base, matching the host:

After creating the jail, configure GPU device access. Edit the bastille jail configuration to allow device access. The key parameters are allow.mlock, sysvshm, and the devfs ruleset entries for nvidia devices. Without mlock, CUDA will fail to pin memory and inference will crash or fall back silently to CPU.

Inside the jail, install Ollama the same way as on the host. The nvidia-smi binary from the host does not need to be reinstalled inside the jail - the kernel module is shared.

pkg install -y bastille
bastille bootstrap 14.2-RELEASE
bastille create ollama-jail 14.2-RELEASE 192.168.1.100

# Edit jail config for GPU access:
cat >> /usr/local/bastille/jails/ollama-jail/jail.conf << 'EOF'
allow.mlock;
allow.sysvipc;
devfs_ruleset = 10;
EOF

# Add nvidia devfs rules to /etc/devfs.rules:
cat >> /etc/devfs.rules << 'EOF'
[devfsrules_jail_nvidia=10]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'nvidia*' unhide
add path 'nvidiactl' unhide
EOF

service devfs restart
bastille start ollama-jail
bastille pkg ollama-jail install -y ollama
bastille sysrc ollama-jail ollama_enable="YES"
bastille service ollama-jail ollama start
// advertisement

Configuring the Ollama Service for Production

Default Ollama config is minimal. For a server that handles multiple users or automated requests, tune three things: the bind address, the model keep-alive timeout, and the number of parallel requests.

Ollama reads environment variables from /etc/rc.conf on FreeBSD when started via the rc script. Set OLLAMA_HOST to expose the service on a specific interface. If you are putting a reverse proxy in front (nginx or relayd are common choices on FreeBSD), keep OLLAMA_HOST bound to localhost and let the proxy handle TLS.

The OLLAMA_KEEP_ALIVE variable controls how long a loaded model stays in VRAM between requests. Default is 5 minutes. For an always-on server, set it to -1 to keep the model loaded indefinitely. This eliminates the 2-4 second cold load penalty on each request.

OLLAMA_NUM_PARALLEL sets concurrent request handling. On a single GPU, 2-4 is the practical max before context switching overhead hurts latency more than it helps throughput.

# In /etc/rc.conf:
sysrc ollama_env="OLLAMA_HOST=127.0.0.1:11434 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=2"

# Or set directly:
cat >> /etc/rc.conf << 'EOF'
ollama_enable="YES"
ollama_env="OLLAMA_HOST=0.0.0.0:11434 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=2 OLLAMA_MAX_LOADED_MODELS=2"
EOF

service ollama restart

Putting nginx in Front of Ollama

Expose Ollama through nginx for TLS termination and basic authentication. Install nginx from pkg, generate a self-signed cert or use certbot for ACME-issued certificates. FreeBSD's certbot package supports DNS challenges which works well when your server is not publicly reachable.

The nginx vhost configuration is straightforward - proxy to localhost:11434 with appropriate headers. If you plan to expose this externally, add HTTP basic auth at minimum. For team use, an API key layer in front is better, but that requires a lightweight proxy like caddy or a custom nginx auth_request setup.

For internal tooling and DevOps automation pipelines, services like taskbotshub.ai can integrate directly with a self-hosted Ollama endpoint using the OpenAI-compatible API, letting you run AI-assisted automation workflows without sending data to external providers.

pkg install -y nginx py39-certbot

# /usr/local/etc/nginx/vhosts/ollama.conf
server {
    listen 443 ssl;
    server_name ai.internal.example.com;

    ssl_certificate /usr/local/etc/ssl/ai.crt;
    ssl_certificate_key /usr/local/etc/ssl/ai.key;

    auth_basic "Ollama API";
    auth_basic_user_file /usr/local/etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
        proxy_buffering off;
    }
}

sysrc nginx_enable="YES"
service nginx start

Monitoring Inference Performance

Track three metrics for a self-hosted LLM server: GPU VRAM utilization, tokens per second (TPS), and request queue depth. FreeBSD does not have a direct equivalent to nvidia-smi dmon, but you can run nvidia-smi in polling mode and pipe it to a log file, or use telegraf with the NVIDIA SMI input plugin.

Ollama itself exposes metrics at the /api/ps endpoint, which shows currently loaded models and their memory usage. For throughput benchmarking, we use a simple shell loop against the REST API and measure wall time.

On our RTX 4070 test setup, llama3.2:3b sustains 47 tokens/sec single-user, dropping to about 28 tokens/sec under two concurrent requests. mistral:7b at 4-bit quantization runs at 22 tokens/sec single-user. These numbers are representative for the hardware class and give you a baseline for capacity planning.

# Watch GPU stats every 2 seconds:
nvidia-smi dmon -s pucvmet -d 2

# Check loaded models:
curl -s http://localhost:11434/api/ps | python3 -m json.tool

# Simple TPS benchmark:
time curl -s http://localhost:11434/api/generate \
  -d '{"model": "llama3.2:3b", "prompt": "Write a 200 word summary of TCP/IP", "stream": false}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'TPS: {d[\"eval_count\"]/d[\"eval_duration\"]*1e9:.1f}')"
// advertisement

Running Open-WebUI as a Frontend

Open-WebUI is the de facto web frontend for Ollama in 2026. It supports model switching, conversation history, document upload, and image generation if you connect it to a diffusion backend. On FreeBSD, the cleanest installation path is via a Python venv since the npm build chain for the frontend can be painful to get running natively.

Alternatively, run Open-WebUI in a Linux jail using FreeBSD's Linux emulation. This is actually reliable in 14.2 and avoids node version conflicts. Install the linux_base-c7 package, enable Linux compatibility, and run the Open-WebUI pip package inside a Linux-compat environment.

For teams setting up internal AI tooling, naming your deployment clearly matters for internal adoption. If you are registering a subdomain or an internal project name for the service, nicename.me is a useful tool for checking whether your chosen project name is available across domains and namespaces before you commit to it in your DNS and SSL certificates.

Open-WebUI connects to Ollama via the API URL. Set OLLAMA_BASE_URL to your Ollama endpoint and WEBUI_SECRET_KEY to a random string. The default port is 8080.

pkg install -y python311 py311-pip
python3.11 -m venv /opt/openwebui
source /opt/openwebui/bin/activate
pip install open-webui

# Create rc script at /usr/local/etc/rc.d/openwebui:
cat > /usr/local/etc/rc.d/openwebui << 'EOF'
#!/bin/sh
# PROVIDE: openwebui
# REQUIRE: ollama
# KEYWORD: shutdown

. /etc/rc.subr

name="openwebui"
rcvar="openwebui_enable"
start_cmd="openwebui_start"
stop_cmd="openwebui_stop"

OLLAMA_BASE_URL="http://127.0.0.1:11434"
WEBUI_SECRET_KEY="changeme-use-a-real-secret"
DATA_DIR="/var/db/openwebui"
PORT=8080

openwebui_start() {
    export OLLAMA_BASE_URL DATA_DIR PORT WEBUI_SECRET_KEY
    /opt/openwebui/bin/open-webui serve \
        --host 127.0.0.1 --port $PORT &
    echo $! > /var/run/openwebui.pid
}

openwebui_stop() {
    kill $(cat /var/run/openwebui.pid)
}

load_rc_config $name
run_rc_command "$1"
EOF

chmod +x /usr/local/etc/rc.d/openwebui
sysrc openwebui_enable="YES"
service openwebui start

Storage Layout and Model Management

Models accumulate fast. A typical deployment with three or four models (3B, 7B, 13B, a code model) uses 25-40GB of disk. Default model storage is /var/db/ollama on FreeBSD. Move this to a dedicated ZFS dataset if you have the disk layout for it - you get compression, easy snapshots before model updates, and quota enforcement.

ZFS lz4 compression on model files gives roughly 1.05-1.1x reduction (GGUF files are already compressed internally), but the real benefit is snapshots. Before pulling a new version of a model, snapshot the dataset so you can roll back if something breaks.

To relocate the model store, stop Ollama, move the data, set OLLAMA_MODELS in the rc.conf environment line, and restart.

# Create a dedicated ZFS dataset:
zfs create -o compression=lz4 -o mountpoint=/opt/ollama-models zroot/ollama-models

# Stop service, move existing models:
service ollama stop
mv /var/db/ollama/models /opt/ollama-models/

# Point Ollama at the new location:
sysrc ollama_env+="OLLAMA_MODELS=/opt/ollama-models"

service ollama start

# Snapshot before pulling a new model:
zfs snapshot zroot/ollama-models@before-llama3.3
ollama pull llama3.3:7b

# List installed models:
ollama list

Troubleshooting Common FreeBSD-Specific Issues

Three issues come up repeatedly when we set up Ollama on FreeBSD systems.

First: Ollama starts but uses CPU despite an NVIDIA GPU being present. The cause is almost always that the nvidia kernel module was not loaded before Ollama started. Check with 'kldstat | grep nvidia'. If it is missing, load it and restart the service. If loading the module fails, check dmesg for PCI device enumeration errors.

Second: JSON parse errors or truncated responses from the API. This usually means the proxy_read_timeout in nginx is too short for the model being used. Increase it to 300s or higher for large models. Also check that proxy_buffering is off - Ollama streams tokens and nginx must pass them through without buffering the full response.

Third: Out of memory kills (OOM) on large models. The FreeBSD kernel's OOM behavior differs from Linux - it tends to kill the process consuming the most memory at the moment of pressure rather than using Linux's oom_score_adj system. Set a ZFS ARC limit to prevent the filesystem cache from competing with GPU VRAM and model weights for RAM: add 'vfs.zfs.arc_max' to /etc/sysctl.conf. On our 32GB system we set it to 8GB, leaving the remainder for Ollama and system overhead.

# Check module load:
kldstat | grep nvidia

# Load manually if missing:
kldload nvidia
kldload nvidia-modeset
service ollama restart

# Limit ZFS ARC to 8GB to prevent RAM contention:
echo 'vfs.zfs.arc_max=8589934592' >> /etc/sysctl.conf
sysctl vfs.zfs.arc_max=8589934592

# Check current ARC usage:
sysctl kstat.zfs.misc.arcstats.size
// advertisement