Systemd is the init system for most modern Linux distributions. Love it or hate it, you need to know it. Here’s how to manage services effectively.

Basic Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Start/stop/restart
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx

# Reload config without restart
sudo systemctl reload nginx

# Enable/disable at boot
sudo systemctl enable nginx
sudo systemctl disable nginx

# Check status
systemctl status nginx

# List all services
systemctl list-units --type=service

# List failed services
systemctl --failed

Writing a Service Unit

Create /etc/systemd/system/myapp.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

[Unit]
Description=My Application
Documentation=https://example.com/docs
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment=NODE_ENV=production
EnvironmentFile=/opt/myapp/.env
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Section Breakdown

[Unit] - Metadata and dependencies

  • Description: Human-readable name
  • After: Start after these units
  • Wants: Soft dependency (start if available)
  • Requires: Hard dependency (fail if not available)

[Service] - How to run

  • Type: simple, forking, oneshot, notify, dbus
  • User/Group: Run as this user
  • ExecStart: Command to start
  • ExecReload: Command to reload
  • Restart: When to restart (always, on-failure, on-abnormal)

[Install] - When to enable

  • WantedBy: Target that pulls in this service

Service Types

Simple (Default)

Process stays in foreground. Systemd considers it started immediately.

1
2
3

[Service]
Type=simple
ExecStart=/usr/bin/myapp

Forking

Traditional daemon that forks. Systemd waits for parent to exit.

1
2
3
4

[Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/bin/myapp -d

Oneshot

Runs once and exits. Good for setup tasks.

1
2
3
4

[Service]
Type=oneshot
ExecStart=/opt/myapp/setup.sh
RemainAfterExit=yes

Notify

Process signals ready via sd_notify. Most reliable for complex apps.

1
2
3

[Service]
Type=notify
ExecStart=/usr/bin/myapp

1
2
3
4

# In your app
import sdnotify
n = sdnotify.SystemdNotifier()
n.notify("READY=1")

Environment Variables

Inline

1
2
3
4

[Service]
Environment=NODE_ENV=production
Environment=PORT=3000
Environment="SECRET_KEY=value with spaces"

From File

1
2
3

[Service]
EnvironmentFile=/opt/myapp/.env
EnvironmentFile=-/opt/myapp/.env.local  # - means optional

Dynamic

1
2

[Service]
ExecStart=/bin/bash -c 'exec /usr/bin/myapp --port=${PORT:-3000}'

Restart Policies

1
2
3
4
5

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5
  • Restart=always: Always restart
  • Restart=on-failure: Only on non-zero exit
  • Restart=on-abnormal: On signal, timeout, or watchdog
  • RestartSec: Wait before restart
  • StartLimitBurst/IntervalSec: Max 5 restarts per 300 seconds

Prevent Restart Loops

1
2
3
4
5
6
7
8

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=500
StartLimitBurst=5

# After hitting limit, wait before allowing more restarts
StartLimitAction=none

Resource Limits

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

[Service]
# Memory limits
MemoryMax=512M
MemoryHigh=400M

# CPU limits
CPUQuota=50%

# File limits
LimitNOFILE=65536

# Process limits
LimitNPROC=4096

# Nice value
Nice=-5

# I/O weight
IOWeight=500

Security Hardening

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

[Service]
# Run as non-root
User=myapp
Group=myapp

# Filesystem protection
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/myapp /var/log/myapp

# Network restrictions
PrivateNetwork=no
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# Capability restrictions
CapabilityBoundingSet=
AmbientCapabilities=

# System call filtering
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

# Prevent privilege escalation
NoNewPrivileges=yes

# Namespace isolation
PrivateDevices=yes
PrivateUsers=yes

Analyze Security

1

systemd-analyze security myapp.service

This scores your service from 0 (most secure) to 10 (least secure).

Logging

Systemd captures stdout/stderr to the journal.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# View logs
journalctl -u myapp.service

# Follow logs
journalctl -u myapp.service -f

# Since boot
journalctl -u myapp.service -b

# Time range
journalctl -u myapp.service --since "1 hour ago"
journalctl -u myapp.service --since "2024-01-15 10:00" --until "2024-01-15 12:00"

# By priority
journalctl -u myapp.service -p err

# JSON output
journalctl -u myapp.service -o json-pretty

Custom Log Configuration

1
2
3
4

[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

Or redirect to files:

1
2
3

[Service]
StandardOutput=append:/var/log/myapp/stdout.log
StandardError=append:/var/log/myapp/stderr.log

Socket Activation

Let systemd handle listening sockets. Service starts on first connection.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# myapp.socket
[Unit]
Description=MyApp Socket

[Socket]
ListenStream=3000
Accept=no

[Install]
WantedBy=sockets.target

1
2
3
4
5
6
7

# myapp.service
[Unit]
Description=MyApp
Requires=myapp.socket

[Service]
ExecStart=/usr/bin/myapp

1
2

sudo systemctl enable myapp.socket
sudo systemctl start myapp.socket

Benefits:

  • Faster boot (services start on demand)
  • Zero-downtime restarts (socket buffers during restart)
  • Reduced resource usage

Timers (Cron Replacement)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# backup.timer
[Unit]
Description=Daily Backup Timer

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=3600

[Install]
WantedBy=timers.target

1
2
3
4
5
6
7

# backup.service
[Unit]
Description=Backup Service

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh

Timer Syntax

1
2
3
4
5
6
7
8
9

OnCalendar=hourly
OnCalendar=daily
OnCalendar=weekly
OnCalendar=*-*-* 04:00:00        # Daily at 4 AM
OnCalendar=Mon-Fri *-*-* 09:00:00 # Weekdays at 9 AM
OnCalendar=*-*-01 00:00:00        # First of month

OnBootSec=5min          # 5 minutes after boot
OnUnitActiveSec=1h      # 1 hour after last activation

1
2
3
4
5

# List timers
systemctl list-timers

# Test calendar syntax
systemd-analyze calendar "Mon-Fri *-*-* 09:00:00"

Template Units

Create multiple instances from one template.

1
2
3
4
5
6

# myapp@.service (note the @)
[Unit]
Description=MyApp Instance %i

[Service]
ExecStart=/usr/bin/myapp --instance %i

1
2
3
4
5
6

# Start specific instances
sudo systemctl start myapp@worker1.service
sudo systemctl start myapp@worker2.service

# Enable at boot
sudo systemctl enable myapp@worker1.service

Specifiers:

  • %i: Instance name (worker1)
  • %I: Unescaped instance name
  • %n: Full unit name
  • %N: Unescaped full unit name

Drop-in Overrides

Override without editing original:

1

sudo systemctl edit nginx.service

Creates /etc/systemd/system/nginx.service.d/override.conf:

1
2

[Service]
Environment=CUSTOM_VAR=value

Or manually:

1
2

sudo mkdir -p /etc/systemd/system/nginx.service.d/
sudo vim /etc/systemd/system/nginx.service.d/limits.conf

1
2

[Service]
LimitNOFILE=65536

1
2

sudo systemctl daemon-reload
sudo systemctl restart nginx

Debugging

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Check unit file syntax
systemd-analyze verify myapp.service

# Show effective configuration
systemctl cat myapp.service
systemctl show myapp.service

# Boot analysis
systemd-analyze blame
systemd-analyze critical-chain myapp.service

# Dependency tree
systemctl list-dependencies myapp.service

Common Patterns

Pre/Post Scripts

1
2
3
4
5
6

[Service]
ExecStartPre=/opt/myapp/pre-start.sh
ExecStart=/usr/bin/myapp
ExecStartPost=/opt/myapp/post-start.sh
ExecStop=/opt/myapp/stop.sh
ExecStopPost=/opt/myapp/cleanup.sh

Watchdog

1
2
3

[Service]
Type=notify
WatchdogSec=30

App must call sd_notify("WATCHDOG=1") within 30 seconds or systemd restarts it.

Graceful Shutdown

1
2
3
4

[Service]
ExecStop=/bin/kill -SIGTERM $MAINPID
TimeoutStopSec=30
KillMode=mixed
  • KillMode=control-group: Kill all processes in cgroup
  • KillMode=mixed: SIGTERM main, SIGKILL rest
  • KillMode=process: Only kill main process

Key Takeaways

  1. Use Type=notify when possible — most reliable startup detection
  2. Set resource limits — prevent runaway processes
  3. Enable security hardening — ProtectSystem, NoNewPrivileges
  4. Use socket activation for zero-downtime restarts
  5. Use timers over cron — better logging, dependencies
  6. Check with systemd-analyze security — aim for low scores

Systemd is powerful. The unit file format is declarative and predictable. Once you learn the patterns, managing services becomes straightforward. 🌍