Systemd unit file: every bit that keeps a Linux service honest

Linux infrastructure work is not only “install packages.” On modern hosts, long-lived work is declared: what starts, in what order, under which identity, how it restarts, and what “healthy” means to the init system.

That declaration lives in unit files. Unit literacy is how efficiency and precision show up as reproducible host behavior. Same Linux honesty whether the host is a lab agent, a license server, or a builder: green/red, restart policy, and logs you can find.

This note fathoms unit files: big picture of types, then one real production unit from a live host—systemd-journald.service as printed by systemctl cat—taken apart and rebuilt from first principles, then how it plugs into sockets, targets, and the rest of the graph.

One-sentence crystal

A systemd unit file is a declarative contract between configuration and PID 1: what resource exists, when it may start, how the process is launched and supervised, and how other units may depend on it—so services are administered as text you can version and explain, not folklore in a shell history.

---

Big picture: what “unit” means

systemd is the init system and service manager on most current Linux distributions. A unit is one managed object: a service process, a timer, a socket, a mount point, a grouping target, and so on.

A unit file is the text (or generated) definition systemd loads to know how to treat that object. Extension ≈ type:

Mental model:

  targets  ── “where we are in boot / runlevel story”
     │
     ├── services  ── “long-lived or oneshot work”
     │      ▲
     │      │ activated / required by
     │      ├── sockets  (listen + often demand-start)
     │      ├── timers   (time start)
     │      └── paths    (fs event start)
     │
     ├── mounts / automounts / swap  ── storage surfaces
     └── slices / scopes  ── resource topology

Most day-to-day service work collapses to: understand a .service, see what .socket / .timer / .path attach to it, place it in a .target graph, and optionally fence it with a .slice.

We start wide. We now narrow—and we use a unit that is already on every serious host.

---

Narrow: the .service workhorse

A service unit answers five first-principle questions:

1. Identity — What is this, and how do humans describe it?
2. Ordering & dependencies — What must exist before this may start? What do we pull in?
3. Execution — Which binary, under which constraints?
4. Supervision — What does “started,” “failed,” and “restart” mean?
5. Install / enablement — How does it attach to boot? (Vendor units often omit a local [Install] because they are pulled in by targets or sockets.)

Those map onto:

Other unit types reuse [Unit] language; only the middle section changes.

---

How to get a real production unit on the page

Do not invent the specimen. Ask the host:

systemctl cat systemd-journald.service

What systemctl cat is doing: it prints the effective unit text systemd uses—vendor file(s) under /usr/lib/systemd/system/ (or /lib/...) plus drop-ins under *.service.d/, in merge order. Comments at the top of each chunk show which path contributed.

That is already a lesson:

Our specimen is journald: the service that owns the journal—the log plane almost every other service depends on via StandardOutput=journal. If you administer Linux infrastructure, you already live downstream of this unit.

---

Specimen: systemd-journald.service (live systemctl cat)

Reconstructed from a real host dump (header abbreviated; drop-in kept). Re-verify on your system.

# /usr/lib/systemd/system/systemd-journald.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#  (license header omitted for length — see file on disk)

[Unit]
Description=Journal Service
Documentation=man:systemd-journald.service(8) man:journald.conf(5)
DefaultDependencies=no
Requires=systemd-journald.socket
After=systemd-journald.socket systemd-journald-dev-log.socket systemd-journald-audit.socket syslog.socket
Before=sysinit.target

# Soft-reboot: avoid SIGKILL corrupting journals (systemd issue #30195).
# Typically soft-reboot.target is not reached; systemd-soft-reboot.service drives soft-reboot.
# Stop journald before soft-reboot service.
Before=soft-reboot.target systemd-soft-reboot.service
Conflicts=soft-reboot.target

# Mount/swap units need journal sockets. Exclude journald + sockets from isolate
# so isolate requests do not tear the logging plane out from under the host.
IgnoreOnIsolate=yes

[Service]
DeviceAllow=char-* rw
ExecStart=/usr/lib/systemd/systemd-journald
FileDescriptorStoreMax=4224
# Services using StandardOutput=journal must not break when journald restarts
FileDescriptorStorePreserve=yes
ImportCredential=journal.*
IPAddressDeny=any
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
OOMScoreAdjust=-250
ProtectClock=yes
Restart=always
RestartSec=0
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_VSOCK AF_INET AF_INET6
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
RuntimeDirectory=systemd/journal
RuntimeDirectoryPreserve=yes
# Audit socket not listed here: this unit can be turned off; link still comes
# from Service= in the socket unit.
Sockets=systemd-journald.socket systemd-journald-dev-log.socket
StandardOutput=null
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM
SystemCallFilter=@system-service
Type=notify-reload
PassEnvironment=TERM
WatchdogSec=3min

# CAP_SYS_PTRACE: needed for /proc/<pid>/exe so journal fields _EXE=/OBJECT_EXE= work.
# Full set may be longer on your host — check systemctl cat.
CapabilityBoundingSet=CAP_SYS_ADMIN CAP_DAC_OVERRIDE CAP_SYS_PTRACE CAP_SYSLOG CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_CHOWN …

# Many split journal files ⇒ many FDs in parallel
LimitNOFILE=524288

# ---------------------------------------------------------------------------
# /usr/lib/systemd/system/systemd-journald.service.d/nice.conf
# Set Nice=-1 to dodge watchdog on soft lockups (LP: #1696970).
# ---------------------------------------------------------------------------
[Service]
Nice=-1

Two files, one effective unit. That is production: vendor base + surgical drop-in.

---

Decompose with first principles

A. [Unit] — graph position for an early service

Description= / Documentation=

• Human name + man pages.
• Principle: identity for operators; docs are part of the contract (man:systemd-journald.service(8), man:journald.conf(5)).

DefaultDependencies=no

• Most units get implicit dependencies (e.g. toward sysinit / basic story). Journald opts out because it must exist very early—before the default dependency set would allow.
• Principle: default dependencies are a safety net for ordinary services; infrastructure that is the early plane must declare the graph by hand.

Requires=systemd-journald.socket

• Hard dependency: journald’s main socket unit must be active. Logging is socket-facing from day zero.
• Principle: Requires= is “nonsensical without this.” For journald, no socket story ⇒ no journal service as designed.

After=…socket … syslog.socket

• Ordering only (plus the Requires above for the main socket). Journald starts after its activation sockets and after syslog.socket so the legacy syslog path is ordered correctly.
• Principle: After= does not pull a unit in by itself; it orders when both are in the transaction. Pair with Wants=/Requires= when you need pull-in.

Before=sysinit.target

• Journald is ordered before the sysinit milestone so “system init” can assume a journal plane.

Soft-reboot stanza (Before= / Conflicts=)

• Comments in the unit are not noise—they encode failure history (journal corruption under soft-reboot + SIGKILL).
• Before=soft-reboot.target systemd-soft-reboot.service — stop/order cleanly
• Conflicts=soft-reboot.target — mutually exclusive with that target
• Principle: production units encode operational scars. Read comments as incident postmortems compressed into config.

IgnoreOnIsolate=yes

• systemctl isolate some.target tears down units not in the new graph. Journald (and its sockets, by design of the stack) must not vanish mid-isolate, or mount/swap and friends lose their logging plane.
• Principle: some units are infrastructure gravity. Isolating them away is self-harm; the unit says so explicitly.

---

B. [Service] — how journald runs

Type=notify-reload

• Not classic simple. Journald participates in systemd’s notify protocol and supports reload semantics (notify-reload): readiness and reload are first-class, not “guess from forking.”
• Principle: Type= is a claim about when start succeeded and how reload works. Wrong type ⇒ wrong supervision.

ExecStart=/usr/lib/systemd/systemd-journald

• Absolute path to the daemon. No shell wrapper.
• Principle: the unit is the source of truth for “what binary is the service.”

Sockets=…

• Binds this service to listed socket units (in addition to the Requires/After graph). Comment notes audit socket is intentionally not listed here so the service can be turned off while socket linkage still lives in the socket unit’s Service= setting.
• Principle: socket↔service links can be declared on either side; read both units before concluding.

Restart=always + RestartSec=0

• Journald is restart-at-once, always. Logging plane must come back immediately.
• Contrast with a CI agent you might run as Restart=on-failure + RestartSec=5s. Principle: restart policy follows failure economics—for journald, downtime is the incident.

WatchdogSec=3min

• systemd expects keepalive notifies within the watchdog window or it treats the service as failed (and restarts per policy).
• Principle: watchdogs are supervision, not decoration—binary must speak the protocol.

StandardOutput=null

• Journald must not log to the journal through the normal “send stdout to journald” path in a naive way—it is the journal. Nulling stdout avoids recursion / nonsense.
• Principle: the logging daemon is a special citizen; copy-paste StandardOutput=journal onto it without thought is a category error.

RuntimeDirectory=systemd/journal + RuntimeDirectoryPreserve=yes

• systemd creates/manages /run/systemd/journal (RuntimeDirectory is under /run). Preserve keeps it across restarts as configured.
• Principle: runtime dirs are part of the unit contract—don’t hand-mkdir folklore when the manager can own the lifecycle.

File descriptor store (FileDescriptorStoreMax= / FileDescriptorStorePreserve=yes)

• Journald parks many FDs (clients, split journals). Preserve across restarts so other services using StandardOutput=journal do not break when journald restarts.
• Principle: production units think about blast radius on neighbors, not only their own process.

Hardening cluster

A dense set of lockdowns—read each as a capability trade:

• Principle: hardening is not “security flavor text.” Each line is a deliberate constraint that can break a feature (e.g. missing CAP_SYS_PTRACE ⇒ missing _EXE= fields). Change drop-ins with a hypothesis and a rollback.

Drop-in: Nice=-1

• From
  /usr/lib/systemd/system/systemd-journald.service.d/nice.conf
  (Ubuntu/LP-style comment: dodge watchdog issues on soft lockups).
• Principle: drop-ins are how distros and admins patch one or two keys without forking the whole vendor unit. systemctl cat exists so you never “fix the unit” while missing *.d/*.conf.

---

C. [Install] — why you may not see it here

• This vendor unit is pulled into early boot by the systemd graph (targets + sockets), not by a classic WantedBy=multi-user.target enable line in the same file.
• Principle: absence of [Install] does not mean “not enabled.” It means enablement is elsewhere (socket units, targets, presets). Use systemctl is-enabled, systemctl list-dependencies, and the socket units—not only the service file.

---

Internalize: pocket map of journald

[Unit]     early boot (DefaultDependencies=no)
           socket Requires/After, Before=sysinit
           isolate + soft-reboot special cases

[Service]  Type=notify-reload + Watchdog
           ExecStart absolute
           Restart=always (logging plane)
           FD store + RuntimeDirectory
           heavy sandbox + CAP ceiling + LimitNOFILE

[drop-in]  Nice=-1 (distro micro-patch)

[Install]  not the main story — graph pulls journald in early

---

Build it back up (mental assembly of the same unit)

You will rarely rewrite journald from empty. The rebuild skill is: could I justify every stanza if I had to re-compose it?

1. Process truth — ExecStart=/usr/lib/systemd/systemd-journald, Type=notify-reload.
2. Activation — sockets first: Requires= + After= + Sockets=.
3. Boot position — DefaultDependencies=no, Before=sysinit.target.
4. Survival — Restart=always, RestartSec=0, WatchdogSec=….
5. Neighbor safety — FD store preserve, IgnoreOnIsolate=yes, soft-reboot ordering.
6. Runtime FS — RuntimeDirectory= (+ preserve).
7. Hardening — add constraints only with a reason (and a man page).
8. Distro drop-in — last millimeter (Nice=-1), never invent a full fork of the vendor file for one key.
9. Verify —
   • systemctl cat systemd-journald.service
   • systemctl status systemd-journald.service
   • journalctl -u systemd-journald -e
   • systemd-analyze verify on units you do author

For your services (agents, tool daemons), the same layers apply with milder restart and less exotic early-boot flags—journald is the extreme teacher.

---

Interconnection: journald and every other unit type

.socket — born attached

• Journald is meaningless without its sockets (systemd-journald.socket, *-dev-log.socket, audit-related units). Socket units own listen FDs; the service consumes them.
• First principle: many production services are socket-activated or socket-required. Always systemctl cat the socket next to the service.

.target — milestones

• Before=sysinit.target places journald under early boot milestones. User services later hang off multi-user.target / default.target. Journald is upstream of almost everything that logs.

.service peers

• Any unit with StandardOutput=journal / StandardError=journal is a client of this plane. Their “works in SSH, silent as a service” bugs often end in journal permissions, rate limits (journald.conf), or disk full on /var/log/journal.

.timer / .path / .mount

• Timers/path units that start oneshots still log through journald once the journal is up.
• Mount units historically care about journal sockets (hence IgnoreOnIsolate rationale)—storage bring-up and logging are coupled in the dependency graph.

.slice

• Journald sits in the system cgroup story; resource pressure that OOM-kills the wrong thing first is why OOMScoreAdjust is tuned. Your builders/agents may use dedicated slices; the logger tries not to die first.

Graph sketch

sysinit.target
   ▲
   │ Before=
systemd-journald.service
   │ Requires / After / Sockets
   ├── systemd-journald.socket
   ├── systemd-journald-dev-log.socket
   └── (audit socket linkage via socket unit Service=)

almost every .service with StandardOutput=journal
   └── streams ──► journald ──► journal files / journalctl

---

Where this meets automate / administer / manage

---

Closing

You have:

1. A map of unit types.
2. A real production specimen—systemd-journald.service via systemctl cat.
3. A first-principles read of early-boot graph, notify/watchdog supervision, FD store, and hardening.
4. The drop-in lesson (nice.conf)—production is merged text.
5. The interconnect story—sockets and targets first, then every logging client.

systemd’s power is not that it has many file extensions. It is that one dependency and supervision model covers the logging daemon itself and the agents you will write tomorrow.

---

Now you know what’s inside a unit file—but have you ever asked where they live in the filesystem?

(Load paths, /usr/lib vs /etc vs /run, drop-in dirs, masks, and why systemctl cat is the microscope—that is the natural next fathom.)