AI Tools That Run on Linux: A Practical Guide for Sysadmins

Local LLM Runtimes: Ollama vs llama.cpp vs LocalAI

Three runtimes dominate local LLM deployment on Linux in 2026: Ollama, llama.cpp, and LocalAI. Each solves a slightly different problem.

Ollama is the fastest path from zero to running model. It packages model management, a REST API on port 11434, and GPU offloading into a single binary. Install it with one command, pull a model, query it. On our test server running Ubuntu 24.04 with CUDA 12.4, we had llama3.1:70b responding via curl in under 10 minutes from a clean install. The daemon runs as a systemd service by default.

llama.cpp is the lowest-level option. You compile it yourself, which means you control BLAS backends, quantization formats, and threading. It supports GGUF models exclusively, but that covers nearly every open-weight model worth running. Use it when you need maximum control over memory layout or when you are embedding inference into a larger C++ application. On CPU-only hardware, llama.cpp with AVX2 and OpenBLAS consistently outperforms Ollama in our benchmarks by 15-20 tokens per second on a 7B model.

LocalAI is an OpenAI-compatible API server. Drop it in front of any GGUF or GGML model and your existing tooling that speaks the OpenAI API protocol works without modification. This matters when you have scripts or internal tools already written against the OpenAI SDK. LocalAI also supports image generation via Stable Diffusion backends and Whisper for transcription, making it the most versatile of the three if you need a unified inference endpoint.

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a model
ollama pull llama3.1:8b
ollama run llama3.1:8b

# Or query the REST API directly
curl http://localhost:11434/api/generate \
  -d '{"model": "llama3.1:8b", "prompt": "List the top 5 iptables rules for a public web server", "stream": false}'

Hardware Requirements and GPU Offloading

Model size in parameters maps directly to VRAM requirements after quantization. A 7B model at Q4_K_M quantization needs approximately 4.5 GB of VRAM. A 13B model needs roughly 8 GB. A 70B model at Q4_K_M needs 40 GB, which means either an A100 80GB, two consumer GPUs with NVLink, or significant CPU RAM fallback.

For CPU-only servers, llama.cpp with Q4_0 quantization on a 7B model is usable at 8-12 tokens per second on a modern 16-core Xeon. That is slow for interactive use but perfectly acceptable for batch tasks like log summarization or generating Ansible playbooks overnight.

Ollama handles multi-GPU automatically when you set the CUDA_VISIBLE_DEVICES environment variable before starting the service. For llama.cpp, the -ngl flag controls how many model layers offload to the GPU. Set it to 99 to push everything onto the GPU; drop it to split the model across GPU and CPU RAM.

On our test server with 128 GB system RAM and no GPU, we ran Mixtral 8x7B at Q5_K_M using llama.cpp with 48 threads and got 6 tokens per second. Slow, but functional for non-interactive use cases. Memory-mapped model files mean the model loads in about 2 seconds after the first cold load, since Linux caches the mapped pages.

# llama.cpp: compile with CUDA support
make LLAMA_CUDA=1 -j$(nproc)

# Run with GPU offloading (all layers to GPU)
./llama-cli \
  -m models/llama-3.1-8b.Q4_K_M.gguf \
  -ngl 99 \
  -c 4096 \
  -p "Generate an nginx config for a reverse proxy with rate limiting"

# Check GPU utilization during inference
watch -n 1 nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv

Code Assistants in the Terminal: avante.nvim, Continue, and Shell Copilot

The useful code assistants for terminal-first workflows fall into two categories: editor plugins that connect to a local or remote LLM, and shell-integrated tools that operate on command history and output.

For neovim, avante.nvim is the current standard. It provides a split-panel chat interface, inline diff application, and context injection from open buffers. Configure it to point at your Ollama instance and you get GPT-4 class assistance without any data leaving your server. The key configuration is setting the provider to 'ollama' and specifying your model in the avante setup call inside init.lua.

Continue.dev works in both VS Code and neovim via its LSP mode. It supports Ollama, LocalAI, and any OpenAI-compatible endpoint. The neovim integration is more limited than avante.nvim but Continue has stronger context management for large codebases, pulling in file tree structure and recently edited files automatically.

For pure shell work, aichat is the most capable CLI tool we tested. It runs on Linux with no GUI dependency, reads piped input, and maintains conversation history per session. Pipe a log file into it, ask it to identify anomalies, get structured output. It connects to Ollama, OpenAI, Anthropic, or any custom endpoint via a YAML config file at ~/.config/aichat/config.yaml.

Shell-gpt (sgpt) is the other strong option. The --shell flag makes it generate and optionally execute shell commands directly. Useful for constructing complex find, awk, or sed pipelines from natural language. We do not recommend the auto-execute mode on production systems, but for workstation use it saves real time.

# Install aichat
cargo install aichat

# Configure for Ollama in ~/.config/aichat/config.yaml
# model: ollama:llama3.1:8b
# Then pipe logs for analysis
tail -n 500 /var/log/nginx/error.log | aichat "Summarize the error patterns and suggest fixes"

# sgpt with shell generation
sgpt --shell "find all files modified in the last 24 hours larger than 100MB in /var"

# avante.nvim minimal config in init.lua
require('avante').setup({
  provider = 'ollama',
  ollama = {
    model = 'llama3.1:8b',
    endpoint = 'http://127.0.0.1:11434',
  },
})

AI Agents for DevOps Automation

AI agents differ from chat assistants in one key way: they execute multi-step tasks autonomously, calling tools, reading output, and adjusting behavior based on results. Several frameworks now run entirely on Linux without cloud dependencies.

CrewAI and AutoGen both run in Python and support local Ollama endpoints as the LLM backend. The practical use case for sysadmins is building agents that can interrogate a system, write a fix, apply it, and verify the result. We built a prototype agent using CrewAI that monitors Prometheus alerts, queries the affected service logs, generates an Ansible task to fix common issues, and opens a pull request in Gitea. The entire pipeline runs on-premises.

For more structured DevOps automation with AI assistance built in, taskbotshub.ai offers pre-built agent workflows targeting infrastructure tasks. Their agents handle incident response playbooks, deployment validation, and configuration drift detection with connectors for Prometheus, PagerDuty, and standard Git workflows. If you need to stand up an AI automation pipeline quickly without building the agent framework yourself, it is worth evaluating as a starting point.

OpenDevin (now OpenHands) is the most capable open-source autonomous agent we tested for software tasks. Running it locally requires pointing it at either a capable local model (70B minimum for useful results) or a cloud LLM. On a server with two A100s running llama3.1:70b, it successfully completed multi-file refactoring tasks and wrote functional Dockerfiles from text descriptions about 70% of the time without human intervention.

For lighter-weight automation, Fabric is a command-line tool that pipes text through predefined or custom LLM patterns. Install it, define patterns as plain text files, and run them against any input. We use it to summarize changelogs before applying updates and to extract action items from long incident postmortems.

# Install CrewAI
pip install crewai crewai-tools

# Minimal agent targeting local Ollama
from crewai import Agent, Task, Crew
from langchain_community.llms import Ollama

llm = Ollama(model="llama3.1:8b", base_url="http://localhost:11434")

analyst = Agent(
    role='Systems Analyst',
    goal='Analyze system logs and identify root causes',
    backstory='Expert Linux sysadmin with 15 years experience',
    llm=llm,
    verbose=True
)

# Install Fabric
pipx install fabric-ai
fabric --setup
# Run a pattern
cat /var/log/syslog | fabric --pattern summarize

Running Models as Systemd Services

Production use of local LLMs requires treating the inference server as a managed service, not a foreground process. Ollama installs its own systemd unit by default. For llama.cpp or LocalAI, you need to write the unit file yourself.

The key systemd configuration decisions are: which user runs the service, GPU access via the right cgroup permissions, and restart behavior. Run inference servers as a dedicated non-root user with access to the video group for GPU access. Set RestartSec to 5 seconds and Restart=on-failure. Cap memory with MemoryMax if the model might exhaust RAM on edge cases.

For LocalAI specifically, it runs as a Docker container or a native binary. The native binary is simpler for systemd integration. Set the working directory to where your models live, expose port 8080, and configure the models directory via the --models-path flag.

Monitor inference performance with Prometheus and the Ollama metrics endpoint at /metrics on port 11434. Grafana dashboards for Ollama exist in the community and give you tokens per second, request queue depth, and model load time. We alert on queue depth over 10 requests as a signal that we need to scale hardware.

# /etc/systemd/system/localai.service
[Unit]
Description=LocalAI inference server
After=network.target

[Service]
Type=simple
User=localai
Group=localai
SupplementaryGroups=video render
WorkingDirectory=/opt/localai
ExecStart=/opt/localai/local-ai \
  --models-path /opt/localai/models \
  --address 0.0.0.0:8080 \
  --threads 8 \
  --context-size 4096
Restart=on-failure
RestartSec=5
MemoryMax=60G

[Install]
WantedBy=multi-user.target

# Enable and start
systemctl daemon-reload
systemctl enable --now localai
journalctl -u localai -f

Embedding AI Into Your Existing Shell Workflows

The highest leverage use of AI on Linux is not a chat interface but integration into existing workflows: cron jobs, shell functions, CI pipelines, and monitoring scripts.

We define a shell function called 'explain' in our .bashrc that takes any command output and explains it in plain language via the Ollama API. Run a complex systemctl output through it. Pipe an strace through it. The latency on a local 8B model is under 2 seconds for most inputs under 1000 tokens.

For CI pipelines, the pattern we find useful is AI-assisted code review as a blocking step. A Python script queries the diff from a merge request, sends it to a local LLM endpoint, and posts the review as a comment. It will not catch all bugs, but it catches obvious issues like missing error handling or hardcoded credentials reliably with a well-prompted model. We run this against a llama3.1:70b model because code review quality on 8B models is too inconsistent for production gating.

Another high-value integration: AI summarization of `journalctl` output during incident response. The query below pulls the last hour of logs from a failing service and returns a structured summary of errors and their frequency. It does not replace reading logs but cuts triage time significantly.

When you are deploying internal AI tooling and need to register a project name or internal subdomain, nicename.me can help you find a clean, available name for your self-hosted inference stack or internal AI platform. Naming matters when the tool will be shared across teams and referenced in documentation.

# Shell function for explaining command output
explain() {
  local input=$(cat)
  curl -s http://localhost:11434/api/generate \
    -d "{\"model\": \"llama3.1:8b\", \"prompt\": \"Explain this system output concisely for a Linux sysadmin:\\n$input\", \"stream\": false}" \
    | jq -r '.response'
}

# Usage
systemctl status postgresql | explain

# Journal summarization during incident response
journalctl -u nginx --since "1 hour ago" --no-pager | \
  aichat "List the top 5 error types, their frequency, and the most recent occurrence timestamp for each"

# AI code review in CI (example GitLab stage)
review_diff:
  stage: review
  script:
    - git diff origin/main...HEAD > /tmp/review.diff
    - python3 scripts/ai_review.py /tmp/review.diff
  allow_failure: true

Security Considerations for Self-Hosted AI

Running AI inference locally removes data from cloud providers, but introduces local attack surface. The three primary risks are: unauthenticated API endpoints, prompt injection through external data sources, and model poisoning via untrusted GGUF files.

Ollama binds to localhost by default. If you change OLLAMA_HOST to bind on a network interface, you expose an unauthenticated HTTP API. Put nginx in front with basic auth or client certificate authentication before exposing Ollama beyond localhost. LocalAI supports API key authentication natively via the --api-keys flag.

Prompt injection is a real risk when your agent pipeline reads external input, such as log files, email, or web content, and passes it to an LLM that can execute tools. Sanitize untrusted input before it reaches the model context. Limit tool permissions for any agent that processes external data. An agent that can only read files should not have a tool that writes to disk.

GGUF model files can theoretically contain malicious payloads, though no public exploits exist as of mid-2026. Download models only from Hugging Face repositories with verified checksums, or from the official Ollama model registry. Verify sha256 checksums before loading any model file.

For multi-tenant inference servers where different teams share an endpoint, implement per-team API keys at the nginx proxy layer and log all requests with the team identifier. Audit logs for unusual query volumes or attempts to extract system prompts.

# nginx reverse proxy with basic auth for Ollama
server {
    listen 443 ssl;
    server_name ai.internal.example.com;

    ssl_certificate /etc/ssl/certs/internal.crt;
    ssl_certificate_key /etc/ssl/private/internal.key;

    location / {
        auth_basic "AI Inference";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:11434;
        proxy_read_timeout 300s;
    }
}

# Verify model checksum before loading
sha256sum llama-3.1-8b.Q4_K_M.gguf
# Compare against the hash published on Hugging Face model card

Benchmarking Your Setup Before Production Use

Before committing hardware to an inference workload, measure actual throughput under load. Single-request latency is not the number that matters. Concurrent request handling at your expected load is what matters.

ollama-benchmark and llm-benchmark are both open-source tools that run configurable concurrent request tests against any OpenAI-compatible endpoint. Run them before deciding on model size and quantization level. On our test server, increasing concurrent users from 1 to 8 dropped per-request throughput from 45 tokens per second to 12 tokens per second on a single RTX 4090. That measurement drove the decision to run two 8B models in parallel rather than one 70B model for our interactive use case.

For llama.cpp, the built-in --bench flag runs an internal benchmark that measures prompt processing speed (pp) and token generation speed (tg) across different batch sizes and context lengths. Run it after compiling with your target BLAS backend to confirm you compiled correctly.

Track memory bandwidth, not just VRAM capacity. LLM inference is memory bandwidth-bound on GPU. An RTX 4090 with 1008 GB/s bandwidth will outperform an A6000 with higher VRAM but lower bandwidth (768 GB/s) on generation speed for equivalent models.

# llama.cpp built-in benchmark
./llama-bench \
  -m models/llama-3.1-8b.Q4_K_M.gguf \
  -p 512 \
  -n 128 \
  -r 3

# Output shows: pp (prompt processing tokens/sec) and tg (generation tokens/sec)
# Example output:
# model    |    size | params | backend | ngl | n_batch | n_ubatch |  pp  |  tg
# llama 8B | 4.66GiB |  8.03B |    CUDA |  99 |     512 |      512 | 1847 |  98

# Load test with wrk against LocalAI endpoint
wrk -t4 -c16 -d30s \
  -s scripts/llm_post.lua \
  http://localhost:8080/v1/completions