systemctl — Command Reference and Operational Patterns

Overview

systemctl is the single client binary for talking to a systemd manager — the system instance running as PID 1, a per-user instance (systemctl --user), or a remote/container instance (--host, --root, -M). It is a thin D-Bus client: almost every verb is translated into method calls on org.freedesktop.systemd1, which is why systemctl works identically over SSH with --host and why a hung manager shows up as a hung systemctl rather than a fast error. Everything it reports comes from the manager’s in-memory state, not from disk — a distinction that matters after manual edits or crashes.

The command’s surface is large but not arbitrary: verbs fall into a small number of functional families (lifecycle, enablement, query, manager-level), and nearly every family shares the same option grammar. Learning the taxonomy beats memorizing flags, because options compose: --now attaches to enablement verbs, --all/--state=/-t filter query verbs, --runtime and --root control where enablement and property changes land.

This page is the operational reference for that surface. Unit-file syntax lives in unit-files.md, per-type directives in service-units.md, the dependency algebra behind list-dependencies in dependency-management.md, and journal plumbing for the debugging recipes in journald.md.

Lifecycle Verbs

Lifecycle verbs submit jobs to the manager’s transaction queue. They are asynchronous by default: systemctl start foo returns once the job is queued, not once the unit reports active — add --wait if a script needs the synchronous form. The verbs and their exact semantics:

Two verbs deserve emphasis. reload picks up a config change for a well-behaved daemon while keeping the process, sockets and connections alive — and its existence is a property of the unit: without ExecReload= it is an error, hence the reload-or-restart fallback. freeze/thaw cgroup-freeze a unit’s processes so you can inspect a wedged or racing daemon without a restart — a recent (v248) feature worth knowing by name.

Enablement Verbs

Enablement verbs manipulate unit files and symlinks on disk — they change what the manager will pull in at boot, not what is running now. The split between “running” and “enabled” is the most common newcomer confusion: enable starts nothing unless --now is given, and start survives no reboot.

is-enabled State Values

systemctl is-enabled foo prints a single token describing the on-disk situation — a compact map of the unit-file loading machinery:

static and generated surprise most people: template units (foo@.service) are static because [Install] symlinks name instances, and every SysV init script reports generated via systemd-sysv-generator — so is-enabled output on a hybrid Debian box mixes both worlds.

Query Verbs

Query verbs are read-only and script-safe; output ranges from human-oriented (status) to machine-oriented (show, JSON with -o json):

show versus status: status renders a curated view, show dumps the property database it is built from. When a service fails, the two properties that name the cause are Result (success, protocol, timeout, exit-code, signal, core-dump, watchdog, start-limit, oom-kill on newer releases) and ExecMainStatus (the numeric exit code or signal). cat shows the merged result of vendor files, admin drop-ins and generator output — it answers “why does the manager think this unit says Restart=always?” in one command.

Manager-Level Verbs

The last family acts on the manager itself rather than individual units:

isolate is the modern telinit: it switches the machine between target trees, and its dangers (stopping everything the new target does not want, wall messages unless --no-wall, AllowIsolate=yes required) are dissected in targets-runlevels.md. set-property is the supported live-tuning path — it persists a drop-in rather than doing raw cgroupfs writes the manager would fight over (see cgroups-resource-control.md).

What daemon-reload Actually Does — and When You Need It

systemctl daemon-reload asks the manager to rebuild its in-memory configuration: it re-reads every unit file and drop-in from all search paths and re-runs all generators (systemd-fstab-generator translating /etc/fstab, systemd-sysv-generator wrapping legacy scripts, plus custom ones). It does not restart, stop or signal any process: running units keep their current configuration until they are next (re)started, and a reload after editing ExecStart= changes nothing about the running daemon.

You need it after any change the manager loads as configuration: editing or adding unit files or drop-ins under /etc/systemd/system or /run/systemd/system; editing /etc/fstab (mount and swap units are generated at load time); adding/removing SysV init scripts or packages that ship unit files (normally handled by package scriptlets, but needed in containers and hand-rolled images). You do not need it for changes the process re-reads itself (an app config file — send systemctl reload instead), for EnvironmentFile contents (read at next start), or for set-property changes (applied immediately).

The failure mode of forgetting a reload is subtle and interview-worthy: the unit runs fine, systemctl cat shows your new drop-in on disk, but the manager still executes the old ExecStart — the next restart then surprises you by applying a config you “changed” hours ago. Configuration management tools model this correctly with a handler that flushes daemon-reload before any restart task.

Key Global Options

Options come before the verb and compose across families:

--no-block matters beyond scripting convenience: a blocking systemctl start from inside a unit’s own ExecStartPre can deadlock the transaction queue — the job cannot complete while the manager waits on you. --no-block (or --job-mode=ignore-dependencies in rescue scenarios) breaks that cycle.

Jobs and –job-mode

Every lifecycle verb becomes a job — a (unit, verb) pair with a state machine (waiting → running → done) — installed into the manager’s transaction. Two jobs on the same unit with the same verb deduplicate; a start and a stop job on the same unit is a collision resolved by the mode:

The queue is visible with list-jobs:

$ systemctl list-jobs
JOB UNIT                                     TYPE  STATE
4471 nginx.service                           start running
4472 network-online.target                   start waiting

2 jobs listed.

A waiting job is blocked behind ordering or an unfinished dependency; a running job is actively starting something. When the transaction engine finds an ordering cycle it breaks it and logs the line you must recognize:

systemd[1]: Breaking ordering cycle by deleting job postgresql.service/start

That means your After=/Before= graph is circular; find the edge you added (usually a drop-in) — the unit whose job was deleted will not start this boot. Full graph mechanics are in dependency-management.md.

systemd-analyze — The Companion Toolkit

systemd-analyze is the profiling and validation sidekick: it queries the same manager (or offline unit directories) and answers the questions systemctl does not.

$ systemd-analyze time
Startup finished in 2.584s (kernel) + 4.851s (initrd) + 1min 3.452s (userspace) = 1min 10.888s
graphical.target reached after 1min 3.391s in userspace

$ systemd-analyze blame | head -3
  13.705s plymouth-quit-wait.service
   7.214s dev-sda1.device
   4.988s networkd-dispatcher.service

$ systemd-analyze critical-chain multi-user.target
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

multi-user.target @1min 3.391s
└─getty.target @1min 3.390s
  └─getty@tty1.service @1min 3.390s +1ms

$ systemd-analyze calendar "Mon *-*-* 02:00:00"
Normalized: Mon *-*-* 02:00:00
    Next elapse: Mon 2025-06-09 02:00:00 CEST
       From now: 3 days left

The classic misreading: blame ranks a slow service top even when it ran in parallel and cost the boot nothing — critical-chain answers “what delayed reaching the default target” and plot settles the argument visually. security ends with a per-unit exposure line (Overall exposure level for nginx.service: 8.5 EXPOSED) — the drop-in below is what lowers that score.

Debugging Recipes

Failed Unit Triage Chain

The standard first minute of a failed-service investigation, in order:

systemctl --failed                            # 1. what is failed at all
systemctl status nginx.service -l --no-pager  # 2. state + last journal lines
journalctl -u nginx.service -b --no-pager     # 3. full boot-scoped log
systemctl show nginx.service -p Result,ExecMainStatus,NRestarts  # 4. failure record
systemctl cat nginx.service                   # 5. effective config (drop-ins included)

Reading step 4: Result=exit-code with ExecMainStatus=1 is an application exit; Result=signal with ExecMainStatus=6 is SIGABRT; Result=timeout means TimeoutStartSec=/TimeoutStopSec= expired; Result=start-limit means StartLimitBurst= was hit and the unit is refused until systemctl reset-failed nginx.service — the classic “I fixed the config but it still won’t start”. Result=watchdog ties into the sd_notify watchdog (apis-development.md).

Stuck Jobs

A systemctl start that hangs is a job stuck in running on some unit in activating state:

systemctl list-jobs                    # which job, which unit
systemctl status <that-unit> -l        # "running (1m 10s / 1m 30s)" = elapsed/timeout

The (elapsed / timeout) pair in status is the deadline countdown. Common blockers: a Requires=-pulled unit that never comes up, network-online.target with systemd-networkd-wait-online.service enabled but no link up (2-minute default timeout), and socket-activated units whose .socket is masked. To unwind the whole transaction, systemctl --job-mode=flush start rescue.target or a targeted systemctl cancel <job-id> clears the queue.

The –user Manager and Linger

systemctl --user is a full parallel manager per UID. Its gotchas are environment-shaped: it needs $XDG_RUNTIME_DIR (usually /run/user/1000), which exists only while the user has a session — over SSH without a login shell, export it manually. Units live in ~/.config/systemd/user/ and /usr/lib/systemd/user/; triage is systemctl --user --failed plus journalctl --user -u myapp. By default everything user-owned dies at logout — loginctl enable-linger alice pins user@1000.service running so her timers and daemons keep working headless, the supported no-root replacement for cron-driven per-user daemons. The manager itself is the system unit user@1000.service, diagnosable with systemctl status user@1000.

A Complete Hardening Drop-In

End-to-end: harden a service without touching the vendor file. Write the drop-in, apply it, verify:

systemctl edit myapp.service

### /etc/systemd/system/myapp.service.d/override.conf
[Service]
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/myapp /run/myapp
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=CAP_NET_BIND_SERVICE
RestrictSUIDSGID=yes
SystemCallFilter=@system-service
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

systemctl daemon-reload                 # drop-in is new configuration
systemctl restart myapp.service         # running process still has old config
systemd-analyze security myapp.service  # verify the exposure score dropped
systemctl show myapp.service -p ProtectSystem,PrivateDevices   # proof

The workflow is the point: edit creates the drop-in (never a vendor-file copy), daemon-reload teaches the manager, restart applies it, security scores the result, and revert undoes everything if the app breaks — the error will be an EPERM in journalctl -u myapp -b. Loosen over-tight filters (SystemCallFilter=@system-service @network-io, MemoryDenyWriteExecute=no for JIT runtimes); the full Protect*/Restrict* catalog lives in service-units.md and systemd.exec(5).

Interview Questions

Q: What does systemctl enable --now foo do, and why is it two operations?

It performs two orthogonal actions: enable writes the [Install]-derived symlinks (typically multi-user.target.wants/foo.service) so the unit is pulled in at every future boot, and --now queues an immediate start job so it is running now. Without --now, enablement takes effect only after reboot; conversely plain start evaporates at next boot. The halves even fail independently — a unit can be enabled but failed, or active but disabled — which is why is-enabled and is-active are separate probes.

Q: How does mask differ from disable, and when is masking the right tool?

disable removes the enablement symlinks, but the unit remains loadable — it can still be started by hand and still pulled in by any other unit that Wants=/Requires= it. mask symlinks the unit name to /dev/null so the manager refuses to load the unit at all, including as a dependency; even an explicit systemctl start fails with “Unit is masked”. Mask is the tool for guarantees: killing a conflicting vendor service (chronyd vs systemd-timesyncd), preventing a socket unit from respawning a service, or holding a unit down across package upgrades. --runtime masks vanish at reboot — the safer default for experiments.

Q: Why is systemctl isolate dangerous?

isolate starts the named target and atomically stops every unit not in that target’s dependency tree. Isolating rescue.target from a multi-user system tears down networking, your SSH daemon (with you attached) and every other service in one transaction; isolating a custom target that does not Want your services stops them silently. It also wall-messages all logged-in users unless --no-wall is passed and requires AllowIsolate=yes on the target. Safe pattern: treat isolate as a maintenance mode switch from a console, and build custom targets deliberately — the design for that is in targets-runlevels.md.

Q: Why does systemd require daemon-reload after editing a unit file, and what exactly does it do?

Unit files are loaded into the manager’s memory; the running process tree is driven from that snapshot, not by re-reading disk. daemon-reload re-reads all unit files and drop-ins from every search path and re-runs all generators (including fstab translation) — but signals no process, so running units keep their old ExecStart until the next (re)start. Forgetting it produces the classic stale-config bug: cat shows your edit, behavior does not change, then a restart applies it all at once. Contrast with systemctl reload, which asks the daemon (via ExecReload) to re-read its own application config without restarting.

Q: You run systemctl restart nginx and it hangs, then fails with a timeout. Walk me through triage.

systemctl status nginx.service -l first: a state of activating (start) with (14s / 1min 30s) shows the start deadline countdown, and Result=timeout after expiry. journalctl -u nginx -b shows whether the process started at all or is waiting on something (a config error, a port held by another process visible in list-sockets, a blocking ExecStartPre). systemctl show -p Result,ExecMainStatus names the failure class exactly. If Result=start-limit, the unit is in refusal mode and systemctl reset-failed is required before the next attempt. The chain — status, journal, show properties, cat the effective unit — resolves nearly every failed-start case in under a minute.

References

• systemctl(1) — the full verb and option reference
• systemd-analyze(1) — profiling, verification and security subcommands
• systemd(1) — the manager itself: what the client talks to
• systemd man page index (latest)
• systemctl(1) — Debian bookworm mirror
• systemd-analyze(1) — Debian bookworm mirror

Cross-References

• Service units — the [Service] section, restart semantics and sandboxing directives the drop-in example used.
• Unit files — search paths, drop-in layering, generators and specifiers behind cat and daemon-reload.
• Journal and journald — the journalctl -u side of every debugging recipe here.
• Targets and the runlevel compatibility layer — what isolate, set-default and the shutdown verbs actually switch.
• Dependencies and ordering — the graph that list-dependencies, jobs and --job-mode operate on.
• systemd hands-on (admin) — everyday command cookbook this page deepens.
• SysVinit tooling — the service/chkconfig/update-rc.d world systemctl replaced.
• Init system comparison — where each family’s control tool stands feature by feature.
• init-systems hub — section overview and reading order.