Host Administration by Agent: Sorting Changes by How Badly They Undo

A bad code change is a revert. A bad change to a running host is a restore, if somebody set one up and tested it.

The work an agent is best at has the worst undo

An agent in a real shell is genuinely good at host administration. It reads manual pages faster than you do, composes the flag combination you always look up, correlates a service failure against three log files at once, and knows the difference between the four ways to make a change persist across a reboot.

It is also, in that same shell, doing work with no version control. A bad code change is a revert. A bad change to a running host is a restore, if somebody set one up. That asymmetry is the whole subject of this article, because the tasks where an agent saves you the most time are exactly the tasks where the mistake is hardest to walk back.

What an agent is actually good at here

Diagnosis, mostly. Given a service that will not start, it will read the unit file, check the journal, notice the permission on the socket, and connect those three facts faster than a person switching between terminals. The work is correlation across many small outputs, which is a strength.

Also: composition. Constructing a correct find invocation, writing the awk that summarises a log, assembling a docker run line with the right mount and user flags. These are tasks where the difficulty is remembering syntax rather than deciding anything, and offloading them is close to pure gain.

Change the config, not the box

The single discipline that makes the rest of this safe. Any change that should survive tomorrow gets made in a file that lives in version control and is then applied — not typed at a prompt against a running system.

This is ordinary infrastructure-as-code advice, and the reason to restate it in an agent context is that the friction which used to enforce it is gone. Editing a config file, committing it and running the apply used to be roughly as much work as making the change directly, so people sometimes did it right. An agent makes the direct change effortless and the disciplined path unchanged, so the ratio has moved and the default drifts toward the box.

Three classes of change, and only one is safe

Reversible. The previous state is recoverable from information that still exists: a config file under version control, a package that can be reinstalled at a pinned version, a container that can be recreated from an image. These are safe to delegate.

Restorable. The previous state exists only in a backup. Safe if and only if the backup is real and someone has tested restoring from it, which is a much smaller set than the set of machines with backups configured.

Unrecoverable. Data that only lived on that machine. A volume that was pruned. A key that was rotated without the old one being kept. A firewall rule that removed your access. These should not be reachable by an agent at all, which is a configuration decision rather than a supervision one — the reasoning is in gating irreversible actions.

Docker: the prune is the one

Container work is a good fit for an agent, with one command that deserves its own rule. A system prune removes stopped containers, unused networks, dangling images and — with the wrong flag — unused volumes. That last category includes the database volume of anything not currently running.

The failure is not exotic. A developer stops a stack to work on something else, an agent tidies up disk space, and the local database from the past eight months is gone. It was never in a backup because it was "just local", which is true right up until it is the only copy of the data you had been testing against.

Deny volume pruning outright. Everything else in the docker command surface is recoverable from an image and a compose file.

systemd: the state that is not in the file

Unit files are text and belong in version control, which makes them the good case. The trap is that a running system's behaviour is not fully described by them: whether a unit is enabled, whether a drop-in overrides part of it, whether something was masked two years ago during an incident.

An agent reading only the unit file will produce a correct-looking explanation of behaviour that does not match reality. Ask for the resolved view — the effective configuration including drop-ins, and the enabled state — rather than the file, and the analysis becomes trustworthy.

The /etc backup that is not a backup

Copying a file to filename.bak before editing it is the universal habit and it protects against exactly one failure: a bad edit noticed immediately. It does not survive the machine, it is not in any inventory, and six months later nobody knows whether the .bak is older or newer than the live file.

It is better than nothing. It is not a reason to consider a change to /etc reversible, and treating it as one is how a machine accumulates configuration nobody can account for.

Package managers and the cascade

Installing one package is rarely one package. A dependency resolver may upgrade a shared library, replace a default implementation, or remove something that conflicts, and the summary scrolls past faster than anyone reads it.

The habit worth insisting on: run the operation in a mode that reports what it will do, have the agent summarise the removals and downgrades specifically, and only then proceed. Removals are the interesting part — an install that removes three packages is a different event from an install that adds three, and the output presents both in the same list.

Firewalls, and the rule that locks you out

The oldest mistake in this domain, and an agent is as capable of it as anyone: a rule that is correct in isolation and removes your own access when applied. On a remote machine that ends the session and the ability to undo it in the same move.

# Schedule the revert BEFORE applying the change. If the new rules lock
# you out, the machine restores itself in ten minutes.
sudo iptables-save > /root/fw-known-good.rules
echo 'iptables-restore < /root/fw-known-good.rules' \
  | sudo at now + 10 minutes

# ... apply the new rules, confirm you still have access ...

# Then, and only then, cancel the revert:
sudo atq            # find the job id
sudo atrm <id>

This pattern generalises well beyond firewalls. Any change that could sever your ability to make further changes should be preceded by a scheduled restoration, cancelled once you have confirmed you still have access. It costs thirty seconds and it is the difference between a mistake and a trip to a data centre.

Demand dry runs, and read them

Most administrative tooling has a mode that reports intended actions without performing them. Making that the default step, rather than an occasional caution, changes the character of the work: the agent proposes, the plan is inspected, the plan is applied.

The value is not only safety. A dry run is also the clearest possible explanation of what a command does, which means the human stays oriented in work they might otherwise let scroll past. That matters over time — delegating administration to an agent while remaining ignorant of your own infrastructure is a trade that feels efficient and compounds badly.

Idempotence is the property to insist on

A script that can be run twice with the same result is safe to retry, safe to run when you are unsure whether it already ran, and safe to leave in a repository for the next person. A script that appends a line to a config file is none of those, and produces a file with the same line eleven times.

Ask for idempotent operations explicitly: check-then-act rather than act, declarative configuration rather than mutation, and writes that replace a managed block rather than appending to a file. An agent will produce either shape and defaults to the imperative one, because that is what most examples look like.

The logs are the real deliverable

For diagnostic work, the valuable output is not the fix. It is the narrative: what was checked, what it showed, what was ruled out. That is the thing a person needs in order to trust the conclusion and to recognise the problem when it recurs.

Ask for it explicitly, because the default answer is a fix with a one-sentence justification. A short account of the investigation costs almost nothing to produce and is the difference between a repaired machine and an understood one.

More than one host is a different problem

Everything above assumes one machine. The moment an agent is running commands across several, the failure modes change: a change applied to eight hosts and not the ninth, a rolling operation that takes down more capacity than intended, a loop that stops halfway and leaves a fleet in two states.

That is a job for configuration management with a convergence model and a concurrency limit, driven by an agent, rather than an agent iterating over hostnames in a shell loop. The distinction is not pedantry: a shell loop has no notion of partial failure, and partial failure across a fleet is the normal outcome rather than the exception.

Reaching other machines from this one

Worth stating because it is easy to miss when reasoning about blast radius. An agent with a shell has your SSH configuration, your agent-forwarded keys and your known hosts, so "this machine" is not the boundary — the boundary is everything this machine can authenticate to without a prompt.

That is a much larger set than most people picture, and it is the subject of ambient authenticated sessions. For host administration specifically, the practical version is to be deliberate about when a key is loaded, and not to leave a forwarded agent live through a long session.

What I would delegate unattended

Diagnosis, freely: read logs, inspect state, correlate, explain. Nothing about it changes the machine and the output is directly useful.

Changes, only where the class is reversible and the change is made in a tracked file and applied. Package installs after a reviewed dry run. Container work with volume operations denied. Nothing on a machine holding data that exists nowhere else, and nothing that could remove my own access without a scheduled revert already in place.

That list is narrower than what an agent is capable of, and it is chosen on recoverability rather than on capability, which is the right axis for work where the undo is a restore.

Ask for the state, not the explanation

A recurring pattern in agent-assisted administration: the agent explains what a system does based on the configuration it read, and the explanation is plausible, internally consistent and wrong, because the running system diverged from its configuration at some point nobody recorded.

The correction is to insist on observed state rather than derived state. Not "what does this unit file say" but "what is actually listening, what is actually enabled, what is actually mounted". Those are different questions and they have different answers on any machine older than a few months.

It is also a good habit for the human. A machine's real configuration is the sum of everything anyone ever did to it, and reading the files gives you the intended version of that history rather than the actual one.

Disk space, and the tidy-up that deletes something

"The disk is full" is one of the most common tasks handed to an agent and one of the most quietly dangerous, because every solution is a deletion and the pressure is to act quickly.

The safe sequence is diagnosis first, in full: what is large, when was it last written, what wrote it. Only then a proposal. The failure mode is an agent that goes straight to the well-known cleanup commands — package caches, old logs, container leftovers — and takes something with them, usually the log somebody needed to diagnose the thing that filled the disk.

Rotate and compress before deleting, move rather than remove where you can, and keep whatever explains the growth. A disk that is full for a reason will be full again next week, and the evidence is what stops that.