Installing Ansible on the Controller Node

Install Ansible from pip into a dedicated virtualenv rather than your system Python. The distro packages on Ubuntu 24.04 and RHEL 9 lag behind upstream by several minor versions, and you will hit bugs that are already fixed. On our test server running Ubuntu 24.04, we use the following pattern for every new controller:

After installation, confirm the version and that the collection path is writable. Ansible looks for collections in ~/.ansible/collections by default, and the ansible-galaxy command will silently fail if that directory is owned by root when you are running as a regular user.

For managed nodes, Ansible requires only Python 3.6+ and an SSH server. No agent, no daemon. If your target fleet mixes RHEL 9, Ubuntu 22.04, and Ubuntu 24.04, the default Python interpreter detection in Ansible 2.17 handles all three without explicit interpreter_python settings in most cases. Set interpreter_python = auto_silent in ansible.cfg to suppress the discovery warning on older playbooks.

python3 -m venv ~/ansible-env
source ~/ansible-env/bin/activate
pip install --upgrade pip
pip install ansible==9.6.0
ansible --version
# ansible [core 2.17.x]
# python version = 3.11.x
ansible-galaxy collection install ansible.posix community.general

Inventory: Static Files, Dynamic Scripts, and Grouping Strategy

Most tutorials show a flat hosts file with five IPs. Real inventories have 10 to 500 hosts across multiple environments, and the grouping strategy you pick in week one becomes load-bearing by week six. We recommend the INI format for static inventories under 50 hosts, and YAML for anything larger because the parent/child group syntax is cleaner.

The directory-based inventory is the correct approach for multi-environment setups. Create an inventory/ directory at the project root, then subdirectories for each environment. Inside each environment directory, put a hosts file and a group_vars/ subdirectory. Ansible merges variables from all matching group_vars files automatically.

For dynamic inventory, the community.general collection includes plugins for AWS EC2, GCP, and bare-metal tools like Foreman. The aws_ec2 plugin queries the AWS API and groups hosts by tag, region, and instance state. Pin the plugin version by pinning the collection version in a requirements.yml file - this prevents a colleague's ansible-galaxy install from pulling a breaking change.

# inventory/production/hosts (YAML format)
all:
  children:
    webservers:
      hosts:
        web01.prod.example.com:
          ansible_user: deploy
        web02.prod.example.com:
          ansible_user: deploy
    dbservers:
      hosts:
        db01.prod.example.com:
          ansible_user: deploy
          ansible_port: 2222
    monitoring:
      children:
        webservers:
        dbservers:

ansible.cfg: The Settings That Actually Matter

The default ansible.cfg is 400 lines of commented-out options. In practice, six to ten settings control 90% of runtime behavior. Place your ansible.cfg in the project root - Ansible loads it before the user-level and system-level configs when you run from that directory.

The forks setting is the most impactful single change you can make. The default is 5, meaning Ansible processes 5 hosts in parallel. On a controller with 4 cores and managed nodes that respond in under 200ms, we found 20 to 30 forks gives a roughly linear speedup with no connection errors on our test infrastructure of 80 nodes. Go above 50 only if you have profiled SSH connection overhead.

Pipelining reduces the number of SSH operations per task from 3 to 1 by keeping the connection open and piping commands through it. It requires requiretty to be disabled in sudoers on managed nodes. On RHEL-family systems, the default sudoers has Defaults requiretty - remove that line or add Defaults !requiretty for the deploy user. With pipelining enabled on our 80-node test cluster, a 15-task playbook dropped from 4m12s to 2m38s.

The callback_whitelist (now callback_enabled in 2.17) line adds useful output without switching to a full callback plugin. timer and profile_tasks are the two to enable first - timer prints total playbook runtime, and profile_tasks prints per-task timing sorted by duration.

[defaults]
inventory = inventory/production
remote_user = deploy
forks = 25
host_key_checking = False
retry_files_enabled = False
stdout_callback = yaml
callback_enabled = timer,profile_tasks
roles_path = roles
collections_path = ./collections

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no

[privilege_escalation]
become = True
become_method = sudo
become_user = root
// advertisement

Writing Playbooks That Do Not Break in Six Months

The most common playbook problem we see in inherited codebases is task-level shell and command modules doing work that purpose-built modules handle idempotently. A shell: task running useradd will fail on the second run when the user already exists unless you add creates: or changed_when: false. Use the ansible.builtin.user module instead - it checks existence before acting and reports changed only when it makes a change.

Always name every task. The auto-generated name from the module and arguments is truncated in output and useless in logs. Use names that describe the intended state, not the action: 'nginx is installed and at version 1.26' is better than 'install nginx'.

Handlers are triggered at the end of a play by default. If you have a task that restarts nginx and then a later task that checks a URL against the running service, the handler has not fired yet. Use meta: flush_handlers to force handler execution at that point in the play. We use this in every deployment playbook that has a smoke-test task.

Variable precedence in Ansible has 22 levels. For practical purposes: extra_vars (-e flag) beats everything, role defaults are overridden by almost everything else, and host_vars beats group_vars. Define your defaults in role defaults/main.yml, environment-specific values in group_vars, and host-specific overrides in host_vars. Never set the same variable in more than two places or you will spend an hour with ansible-inventory --host to debug why a node is getting the wrong value.

---
- name: Deploy nginx on webservers
  hosts: webservers
  gather_facts: true

  handlers:
    - name: nginx is reloaded
      ansible.builtin.service:
        name: nginx
        state: reloaded

  tasks:
    - name: nginx package is installed at target version
      ansible.builtin.package:
        name: nginx=1.26.*
        state: present
      notify: nginx is reloaded

    - name: nginx main config is deployed from template
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        group: root
        mode: '0644'
        validate: /usr/sbin/nginx -t -c %s
      notify: nginx is reloaded

    - name: pending handlers are flushed before smoke test
      ansible.builtin.meta: flush_handlers

    - name: nginx responds on port 80
      ansible.builtin.uri:
        url: http://localhost/healthz
        status_code: 200
      register: healthcheck
      retries: 3
      delay: 5
      until: healthcheck.status == 200

Roles: Structure, Naming, and Galaxy

Roles are the unit of reusable Ansible code. A role named nginx_proxy should do exactly that: configure nginx as a reverse proxy. It should not also manage firewall rules or install certbot. Keep roles single-purpose, and you will reuse them across projects.

Create roles with ansible-galaxy role init rather than mkdir - it generates the correct directory skeleton including defaults, vars, tasks, handlers, templates, files, meta, and tests. The meta/main.yml file matters if you publish to Galaxy or share internally, because it declares dependencies on other roles that Ansible resolves before running the role.

When naming roles that get shared across teams or published, short and unambiguous names help. The same principle applies to naming Ansible projects and Git repositories in general - clarity over cleverness. For registering a domain for an internal Ansible automation portal or GitOps dashboard, nicename.me is a registrar we have used for clean .dev and .io domains with straightforward WHOIS privacy.

For role dependency management in a team environment, use a requirements.yml file and pin role versions. Run ansible-galaxy install -r requirements.yml --roles-path ./roles at the start of your CI pipeline. This gives you reproducible installs. Without version pinning, a Galaxy role update at 2 AM can break your 6 AM deployment.

# Create a new role skeleton
ansible-galaxy role init roles/nginx_proxy

# requirements.yml - pin everything
---
roles:
  - name: geerlingguy.docker
    version: 7.1.0
  - name: geerlingguy.nodejs
    version: 7.0.0

collections:
  - name: community.general
    version: '>=9.0.0,<10.0.0'
  - name: ansible.posix
    version: '>=1.5.0'

# Install from requirements
ansible-galaxy install -r requirements.yml --roles-path ./roles
ansible-galaxy collection install -r requirements.yml -p ./collections

Ansible Vault: Encrypting Secrets Without Losing Your Mind

Ansible Vault encrypts files and individual variables using AES-256. Never store plaintext passwords, API keys, or TLS private keys in a playbook or variable file that goes into version control. The vault password itself should come from a file outside the repository or from an environment variable, not typed interactively in CI.

The recommended pattern in 2026 is to use vault-encrypted variable files alongside plain variable files. Name the encrypted file vault.yml and the plain file vars.yml. In vars.yml, reference the vault variable: db_password: '{{ vault_db_password }}'. The vault.yml file contains vault_db_password: 'actualpassword' and is encrypted. This way, grep on the unencrypted files reveals the variable names without exposing values.

For CI/CD pipelines, store the vault password as a CI secret and write it to a temp file at pipeline start. Pass --vault-password-file to ansible-playbook. In GitLab CI, the pattern looks like echo "$ANSIBLE_VAULT_PASSWORD" > /tmp/vault_pass && ansible-playbook site.yml --vault-password-file /tmp/vault_pass.

To rotate a vault password across all encrypted files in a repository, use ansible-vault rekey with the --new-vault-password-file flag. Do this quarterly or whenever someone with vault access leaves the team. We keep a Makefile target called make rekey-vault that handles the file list automatically.

# Encrypt an existing file
ansible-vault encrypt group_vars/all/vault.yml

# Edit an encrypted file in place
ansible-vault edit group_vars/all/vault.yml

# Encrypt a single string (inline variable)
ansible-vault encrypt_string 'MyS3cr3tP@ss' --name 'vault_db_password'

# Run playbook with vault password from file
ansible-playbook site.yml --vault-password-file ~/.vault_pass

# Rekey all vault files after password rotation
find . -name 'vault.yml' -exec ansible-vault rekey \
  --vault-password-file ~/.vault_pass_old \
  --new-vault-password-file ~/.vault_pass_new {} \;
// advertisement

Tags, Limits, and Running Subsets of a Playbook

Tags let you run a subset of tasks without editing the playbook. Apply tags at the task, block, or role level. Consistent tag names across your entire playbook tree pay off when you need to push only the configuration-file tasks across 80 hosts without reinstalling packages.

We use a three-tier tag taxonomy on our projects: component (nginx, postgresql, docker), action (install, configure, restart), and environment (staging, production). Any task that touches a config file gets both its component tag and the configure tag. Running --tags configure,nginx then runs only nginx config tasks.

The --limit flag narrows execution to a subset of hosts from the inventory. It accepts a host name, group name, pattern, or a file with a leading @. The @ syntax is useful after a failed run - Ansible writes a retry file listing the failed hosts, and you can rerun with --limit @site.retry.

Combine --check mode with --diff to audit what would change before applying. Check mode does not execute anything that would modify state, but it does gather facts, which means it still makes SSH connections and runs setup. On a 50-host inventory, check mode with diff adds about 30 seconds versus a real run on our infrastructure.

# Run only nginx configuration tasks in production
ansible-playbook site.yml --tags nginx,configure --limit webservers

# Dry run with diff output
ansible-playbook site.yml --check --diff --limit web01.prod.example.com

# Rerun after partial failure using retry file
ansible-playbook site.yml --limit @site.retry

# List all tasks that would run for a given tag
ansible-playbook site.yml --tags configure --list-tasks

# Run against a pattern: all hosts in webservers except web03
ansible-playbook site.yml --limit 'webservers:!web03.prod.example.com'

Testing Playbooks with Molecule

Molecule is the standard framework for testing Ansible roles. It spins up containers or VMs, runs your role, and then runs a verifier (default is Ansible itself, or you can use Testinfra with pytest). We run Molecule in Docker on every pull request using GitHub Actions.

Install Molecule with the Docker driver in your virtualenv: pip install molecule molecule-plugins[docker]. Initialize a Molecule scenario inside an existing role with molecule init scenario -d docker. This creates a molecule/default/ directory with a converge.yml playbook that applies your role to a fresh container and a verify.yml for assertions.

Write verify tasks that test the actual state, not just that the role ran. Assert the service is running, the config file contains the expected values, and the port is open. We use ansible.builtin.stat, ansible.builtin.command with register, and community.general.listen_ports_facts in our verify playbooks.

For teams building heavier automation stacks and looking at AI-assisted testing or infrastructure drift detection, taskbotshub.ai covers tooling in that space worth evaluating alongside Molecule for GitOps workflows.

Run the full test sequence with molecule test, which destroys any existing instance, creates a new one, runs converge, runs verify, and destroys again. Use molecule converge during development to iterate without the create/destroy overhead.

pip install molecule 'molecule-plugins[docker]' pytest testinfra

# Initialize molecule in an existing role
cd roles/nginx_proxy
molecule init scenario -d docker

# Iterate during development
molecule converge
molecule verify

# Full test including destroy
molecule test

# molecule/default/verify.yml example
---
- name: Verify nginx_proxy role
  hosts: all
  tasks:
    - name: nginx service is running
      ansible.builtin.service_facts:

    - name: assert nginx is active
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'

    - name: port 80 is listening
      ansible.builtin.wait_for:
        port: 80
        timeout: 5

Performance Tuning for Large Inventories

When your inventory grows past 100 hosts, three settings have the most impact: forks, pipelining, and fact caching. We covered forks and pipelining in the ansible.cfg section. Fact caching eliminates the setup module run on every play for hosts whose facts you have already collected.

Enable fact caching with jsonfile or redis. The jsonfile backend writes JSON to a directory and is zero-dependency. Redis is better for multi-controller setups or when you want to query facts from outside Ansible. Set fact_caching_timeout in seconds - we use 3600 (one hour) for stable fleets where hardware does not change often.

The strategy plugin controls how Ansible moves through hosts and tasks. The default linear strategy runs all hosts through task 1 before any host starts task 2. The free strategy lets each host advance through tasks as fast as it can, which is faster overall but produces interleaved output that is harder to read. Use free on long plays with many tasks across heterogeneous hosts. Set it per-play with strategy: free rather than globally.

For very large playbooks, profile which tasks consume the most time using the profile_tasks callback. In our experience, template rendering and package installation dominate runtime. You can parallelize across host groups using async tasks with poll: 0 for package installs, then use ansible.builtin.async_status to wait for completion. This is advanced and adds complexity - reach for it only when profile_tasks shows a single slow task blocking a large host group.

# ansible.cfg additions for large inventories
[defaults]
forks = 30
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_fact_cache
fact_caching_timeout = 3600

# Per-play strategy override
- name: Install packages on all nodes
  hosts: all
  strategy: free
  tasks:
    - name: baseline packages are installed
      ansible.builtin.package:
        name:
          - vim
          - curl
          - htop
          - jq
        state: present

# Async package install example
    - name: docker is installing asynchronously
      ansible.builtin.package:
        name: docker-ce
        state: present
      async: 300
      poll: 0
      register: docker_install_job

    - name: wait for docker install to finish
      ansible.builtin.async_status:
        jid: '{{ docker_install_job.ansible_job_id }}'
      register: job_result
      until: job_result.finished
      retries: 30
      delay: 10
// advertisement

Error Handling and Partial Failure Recovery

Ansible stops a play for a host when a task fails on that host by default. It continues with remaining hosts. The max_fail_percentage play option lets you abort the entire play if failures exceed a threshold - critical for rolling deploys where you do not want to take down more than 20% of a pool.

The block, rescue, and always structure gives you try/catch/finally semantics inside a play. Put the risky tasks in block, recovery tasks in rescue (which runs only if block fails), and cleanup in always (which runs regardless). This is the correct pattern for tasks like database schema migrations where a failure should trigger an automated rollback procedure.

The ignore_errors: true option should be used sparingly and only when you genuinely do not care about failure - for example, running a smoke test before any changes and recording the baseline state. If you use ignore_errors on a task that modifies state, you will get hard-to-debug partial configurations. Use failed_when instead to define what actually constitutes failure based on the registered output.

For rolling updates across a pool, use serial to limit how many hosts run at once. serial: '20%' runs the play on 20% of the inventory at a time, completing all tasks on that batch before moving to the next. Combine with max_fail_percentage: 0 to abort the rolling update immediately on any failure.

- name: Database migration with rollback
  hosts: dbservers
  serial: 1
  max_fail_percentage: 0

  tasks:
    - block:
        - name: migration script runs
          ansible.builtin.command:
            cmd: /opt/app/bin/migrate up
          register: migration_result
          failed_when:
            - migration_result.rc != 0
            - "'already at latest' not in migration_result.stdout"

        - name: application service is started
          ansible.builtin.service:
            name: myapp
            state: started

      rescue:
        - name: migration is rolled back on failure
          ansible.builtin.command:
            cmd: /opt/app/bin/migrate down 1

        - name: failure is logged to monitoring
          ansible.builtin.uri:
            url: https://alerts.example.com/ansible/failure
            method: POST
            body_format: json
            body:
              host: '{{ inventory_hostname }}'
              playbook: '{{ ansible_play_name }}'

      always:
        - name: migration log is archived
          ansible.builtin.fetch:
            src: /var/log/myapp/migration.log
            dest: ./logs/{{ inventory_hostname }}/
            flat: false