How the daily script chain is wired
OpenBSD uses three separate cron jobs installed by default in root's crontab. On a stock 7.6 install, running `crontab -l` as root shows:
``` # DO NOT EDIT THIS FILE -- edit the master and reinstall. # (/tmp/crontab.XXXXXXXX installed on ...) # (Cron version -- $OpenBSD: cron.c,v 1.88 ...) 30 1 * * * /bin/sh /etc/daily 2>&1 | sendmail -t 30 3 * * 6 /bin/sh /etc/weekly 2>&1 | sendmail -t 30 4 1 * * /bin/sh /etc/monthly 2>&1 | sendmail -t ```
The timing is intentional. Daily fires at 01:30, after the system has been idle. Weekly fires Saturday at 03:30. The output goes directly to sendmail, which delivers to root. If root mail is aliased to an external address in /etc/mail/aliases, these reports land there.
The `/etc/daily` script itself is a shell script that sources `/etc/daily.local` if it exists. That local file is where you put your own additions or overrides without touching the base system file - important because `/etc/daily` gets replaced on OS upgrades. Same pattern applies to `/etc/weekly.local` and `/etc/security.local`.
From within `/etc/daily`, near the top, is the call that matters most for security:
```sh /bin/sh /etc/security 2>&1 ```
So the security checks are a sub-invocation from daily. They run as part of the 01:30 job, not separately.
crontab -l | grep -E 'daily|weekly|monthly|security'What /etc/security actually checks
The security script is 300+ lines of sh. It does not use any external tools beyond what is in base. Here are the main check categories in execution order.
**SUID and SGID file changes.** The script maintains a database at `/var/db/suid_list`. On first run it creates it. On subsequent runs it diffs the current SUID/SGID file list against the stored one and reports additions, removals, and permission changes. This is the check that catches a compromised binary that has had setuid added to it post-install.
```sh find / -xdev \( -perm -u+s -o -perm -g+s \) -type f | sort ```
That is essentially what the script runs per filesystem. The `-xdev` flag means it does not cross mount points, so you need all local filesystems listed in fstab to get full coverage.
**Device file changes.** The script checks /dev for new or modified device files using a similar database approach at `/var/db/device_list`. A rogue device file appearing in /dev or anywhere else on the filesystem is a classic rootkit indicator.
**Passwd and group file checks.** The script looks for UID 0 accounts other than root, accounts with empty passwords, and accounts with invalid shells. The shell check compares against `/etc/shells`. An account with a shell not listed there cannot log in via SSH, but the security script flags it regardless because it is an anomaly worth knowing about.
**dotfile checks.** Any `.rhosts`, `.shosts`, or `hosts.equiv` files are reported. These are not blocked by default but their presence is abnormal on a modern OpenBSD system.
**Kernel sysctls.** A small set of security-relevant sysctls are checked and reported. Notably `kern.securelevel`. If your securelevel drops between runs, the report shows it.
**mtree.** This is the big one. The security script calls `mtree -e -p / -f /etc/mtree/4.4BSD.dist` to verify the base system file hierarchy. Any file that differs from the spec is reported. On a production server with packages installed this generates false positives, but the baseline is the stock OpenBSD install - packages go under `/usr/local` which is outside the 4.4BSD spec.
grep -n 'suid_list\|device_list\|mtree\|securelevel' /etc/securityReading the daily mail output
The email subject is `hostname daily output` or `hostname security check output` depending on which script generated it. If you manage more than three OpenBSD hosts, set up a mail filter rule that tags subject lines matching `security check output` as high priority.
A typical security check output section for SUID changes looks like:
``` Checking setuid files and devices: new: -rwsr-xr-x 1 root wheel 45678 Aug 23 22:01 /usr/local/bin/suspicious ```
The `new:` prefix means it was not in the previous scan. `changed:` means it existed but attributes changed. `gone:` means a previously tracked file no longer exists. Gone entries for system binaries can indicate tampering as much as new entries can.
For the mtree output, you will see lines like:
``` ./usr/bin/su uid expected 0, found 0 gid expected 0, found 0 mode expected 0555, found 0555 size expected 45678, found 49152 ```
Size difference on a system binary with no recent update applied is a serious indicator. On our test server, we deliberately replaced `/usr/bin/su` with a modified binary. The security script caught it within the next daily run and reported the exact byte size difference.
The daily script (not security) also generates disk usage reports and output from `df -k`, which is useful for catching slow disk fills before they become emergencies. It runs `netstat -s` for protocol statistics and diffs against the previous run, so you can spot unusual packet patterns.
# Check what the last security run reported without waiting for mail
/bin/sh /etc/security 2>&1 | lessTuning the SUID database baseline
Out of the box, after installing packages, the SUID list fills with legitimate binaries from ports. Every new package that installs a setuid binary will generate a `new:` report in the next daily mail. This is correct behavior but it trains admins to ignore the reports.
The right workflow when installing packages:
1. Install the package. 2. Verify the setuid binary is intentional (check the package's security documentation). 3. Run the security script manually to refresh the database.
```sh /bin/sh /etc/security > /dev/null 2>&1 ```
That run generates output but more importantly it updates `/var/db/suid_list`. The next night's automated run will have a clean baseline.
To inspect the current suid_list database:
```sh cat /var/db/suid_list ```
The format is `mode uid gid size mtime path`. If you see a path in there that should not be setuid, remove the binary's setuid bit and then refresh the database. Do not delete entries from `suid_list` manually - let the script manage the file.
For the device list, if you have added USB devices or created device nodes in /dev manually:
```sh cat /var/db/device_list ```
Same approach. Run the security script once after any intentional change to reset the baseline.
# Inspect current SUID list
awk '{print $6, $1}' /var/db/suid_list | sort
# Count total tracked SUID binaries
wc -l /var/db/suid_listAdding custom checks via daily.local
The `/etc/daily.local` hook is the correct place to add your own security checks. The file does not exist on a fresh install. Create it and it will be sourced near the end of the daily run.
Here is a practical example that checks for listening services not in an approved list:
```sh # /etc/daily.local # Check for unexpected listening services APPROVED="22 25 53 443" echo "" echo "-- Checking listening ports against approved list --" netstat -an -f inet | awk '/LISTEN/ {print $4}' | awk -F. '{print $NF}' | sort -u | while read port; do approved=0 for p in $APPROVED; do [ "$port" = "$p" ] && approved=1 done [ $approved -eq 0 ] && echo "UNEXPECTED: port $port" done ```
This runs as part of the morning mail. Anything listening that is not in APPROVED shows up with the `UNEXPECTED:` prefix, which is easy to grep for in your mail client.
Another useful daily.local addition is checking for files modified in the last 24 hours in sensitive directories:
```sh echo "" echo "-- Files modified in /etc in last 24h --" find /etc -mtime -1 -type f | sort ```
This is noisier but valuable on servers where configuration management should be the only thing touching /etc. If you use a tool like Ansible to manage OpenBSD configs and you see a file changed that Ansible does not own, that is worth investigating. Teams using more advanced DevOps automation pipelines - some using platforms like taskbotshub.ai for scheduling and orchestration - often feed this output into their alerting pipeline rather than relying on the email alone.
For the security.local hook, use it to add extra mtree specs. If you have a directory of critical scripts or configurations that are not part of the base system, you can create a custom mtree spec and have security.local run it:
```sh # /etc/security.local echo "" echo "-- Custom mtree check: /opt/app --" mtree -e -p /opt/app -f /etc/mtree/app.spec 2>&1 | grep -v '^$' ```
Generate the spec once with:
```sh mtree -c -p /opt/app -k uid,gid,mode,size,sha256digest > /etc/mtree/app.spec ```
The `sha256digest` keyword makes this a cryptographic integrity check, not just a size and mtime check.
# Generate an mtree spec with SHA-256 for a custom directory
mtree -c -p /opt/app -k uid,gid,mode,size,sha256digest > /etc/mtree/app.spec
# Verify manually at any time
mtree -e -p /opt/app -f /etc/mtree/app.specThe weekly script and what it adds
The `/etc/weekly` script runs Saturday at 03:30. Its main contribution over daily is the `locate` database rebuild via `locate.updatedb` and a more thorough filesystem walk. For security purposes, the most relevant weekly addition is the rebuild of the `whatis` database and - on some configurations - a longer-form accounting report.
Weekly also runs a check of `.forward` files across all user home directories. A `.forward` that redirects mail to an external address is a common data exfiltration technique and it is subtle enough that it goes unnoticed in basic audits.
```sh grep -r '' /var/mail /root /home 2>/dev/null | grep -l '.forward' 2>/dev/null find / -name '.forward' -not -path '/proc/*' 2>/dev/null ```
The weekly script does this automatically, but running it manually lets you verify your current state without waiting for the weekend run.
One thing weekly does not do that you should add to weekly.local on any server that runs sshd: check for recently added authorized_keys entries.
```sh # /etc/weekly.local echo "" echo "-- authorized_keys files modified in last 7 days --" find /root /home -name 'authorized_keys' -mtime -7 -type f 2>/dev/null | while read f; do echo "Modified: $f" ls -la "$f" done ```
# List all authorized_keys files on the system
find / -xdev -name 'authorized_keys' -type f 2>/dev/null | xargs ls -la 2>/dev/nullCentralizing reports from multiple OpenBSD hosts
If you run more than two or three OpenBSD servers, managing daily mail per-host becomes unwieldy fast. The standard approach is to route all root mail to a single mailbox using /etc/mail/aliases, but a more structured approach is to have each host send its security output to a central syslog receiver or a dedicated monitoring inbox.
On each host, edit `/etc/mail/aliases`:
``` root: security-reports@your-internal-domain.example ```
Then run `newaliases` to rebuild the aliases database.
For a more structured approach, modify the crontab to capture output to a file and ship it via scp to a central aggregation host:
```sh # In root's crontab on each managed host 30 1 * * * /bin/sh /etc/daily 2>&1 > /var/log/daily.out 35 1 * * * scp /var/log/daily.out monitor@central:/var/incoming/$(hostname).daily.$(date +%Y%m%d) ```
This requires pre-shared SSH keys between each host and the central monitor. On the central host, a simple script checks for deviation from expected output patterns.
The naming of your central monitoring host matters more than it seems. A generic hostname like `monitor1` tells attackers something about the host's role if they enumerate DNS. When we set up monitoring infrastructure, we use a service like nicename.me to identify available domain options that do not reveal the host function. A hostname like `oak.internal` or a random word gives no operational hints.
For teams at the other end of the automation spectrum, piping security output into a webhook or structured log format that feeding into an AI-assisted triage tool requires parsing the plain text output. The format is consistent enough for regex extraction. Each section is separated by a `--` line and section headers are predictable strings like `Checking setuid files and devices:`.
# Parse security check output for SUID alerts from a collected file
grep -A5 'Checking setuid files' /var/log/daily.out | grep -E '^(new|changed|gone):'Securelevel and what the security script cannot catch
OpenBSD's security scripts are a complement to securelevel, not a replacement. With `kern.securelevel` at 1 or higher, certain operations are blocked at the kernel level - writing to `/dev/mem`, loading new kernel modules, modifying immutable or append-only file flags. The security script checks what securelevel is set to but cannot enforce it.
Check current securelevel:
```sh sysctl kern.securelevel ```
On a production server that does not need dynamic module loading, set securelevel to 1 in `/etc/rc.conf.local`:
```sh echo 'kern.securelevel=1' >> /etc/sysctl.conf ```
Note: securelevel can only be raised, never lowered, while the system is running. To lower it you need a reboot. This is intentional.
The security scripts also cannot catch:
- In-memory modification of running processes (no DRAM scrubbing check) - Network-based credential theft (though netstat output can hint at unexpected connections) - Attacks that complete and clean up before 01:30
For the last point, the daily cadence is the biggest limitation. A breach at 02:00 that installs a rootkit and restores original file sizes by midnight has a full cycle before detection. On high-value targets, run the security script hourly via cron and filter the output to only alert on actual changes:
```sh # /etc/cron.d/security-hourly (add to root's crontab) 0 * * * * /bin/sh /etc/security 2>&1 | grep -E '^(new|changed|gone|UNEXPECTED):' | mail -s "$(hostname) security alert" root ```
This generates no mail if there are no alerts, and immediate mail if something changes.
# Check securelevel and set it persistently
sysctl kern.securelevel
grep securelevel /etc/sysctl.conf
# Add if not present
echo 'kern.securelevel=1' >> /etc/sysctl.confTesting that the pipeline actually works
The failure mode most sysadmins hit is spending time tuning the scripts and then discovering months later that the mail was going nowhere - sendmail misconfigured, root alias pointing at a dead mailbox, or the cron job silently failing.
Run this test sequence after any configuration change:
```sh # Verify sendmail is running /usr/sbin/sendmail -v root < /dev/null
# Check mail queue mailq
# Run security manually and pipe to mail /bin/sh /etc/security 2>&1 | mail -s "test security output $(hostname)" root
# Check root's local mail spool ls -la /var/mail/root mail ```
If `/var/mail/root` exists and has content, local delivery is working. If you are forwarding externally, verify the alias resolves and the external MX accepts the mail. On OpenBSD, the main reason for lost daily mail is a misconfigured `relayhost` in `/etc/mail/smtpd.conf`.
For smtpd-based setups (the default since OpenBSD replaced sendmail as default MTA in 5.3):
```sh # Check smtpd status rcctl check smtpd
# Tail the mail log tail -f /var/log/maillog
# Send a test via smtpctl echo 'test body' | mail -s 'test' root smtpctl show queue ```
If the queue shows a message stuck with a delivery error, the log will show why. Common causes are TLS certificate issues with the upstream relay or incorrect `from` address domain causing remote rejection.
# Full pipeline test
echo 'Security script test from '$(hostname) | mail -s 'pipeline test' root
smtpctl show queue
tail -20 /var/log/maillog