Systemd services look simple until they don’t start, restart unexpectedly, or fail silently. Here’s how to write service files that work reliably in production.
Basic Structure#
1
2
3
4
5
6
7
8
9
10
11
12
| # /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/myapp
Restart=always
[Install]
WantedBy=multi-user.target
|
Three sections, three purposes:
- [Unit] - What is this, what does it depend on
- [Service] - How to run it
- [Install] - When to start it
Service Types#
Type=simple (Default)#
1
2
3
| [Service]
Type=simple
ExecStart=/usr/bin/myapp
|
Systemd considers the service started immediately when ExecStart runs. Use when your process stays in foreground.
Type=exec#
1
2
3
| [Service]
Type=exec
ExecStart=/usr/bin/myapp
|
Like simple, but waits for the binary to actually execute (not just fork). Better for catching startup failures.
Type=forking#
1
2
3
4
| [Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/bin/myapp --daemon
|
For traditional daemons that fork and exit. Parent exits, child continues. Requires PIDFile.
Type=oneshot#
1
2
3
4
| [Service]
Type=oneshot
ExecStart=/usr/bin/setup-script.sh
RemainAfterExit=yes
|
For scripts that run once. RemainAfterExit=yes keeps the service “active” after completion.
Type=notify#
1
2
3
| [Service]
Type=notify
ExecStart=/usr/bin/myapp
|
Service explicitly notifies systemd when ready via sd_notify(). Best for apps that support it.
Dependencies and Ordering#
After vs Requires#
1
2
3
4
5
6
7
8
9
| [Unit]
# Start after these (ordering)
After=network.target postgresql.service
# Require these to be active (dependency)
Requires=postgresql.service
# Weaker dependency - don't fail if missing
Wants=redis.service
|
After = ordering only (start after X)
Requires = hard dependency (fail if X fails)
Wants = soft dependency (nice to have)
Network Dependencies#
1
2
3
4
5
6
7
| [Unit]
# Wait for network stack
After=network.target
# Wait for network to be fully online
After=network-online.target
Wants=network-online.target
|
Use network-online.target when you need actual connectivity, not just interfaces up.
Database Dependencies#
1
2
3
4
5
6
7
8
| [Unit]
Description=App that needs PostgreSQL
After=postgresql.service
Requires=postgresql.service
# Or for socket activation
After=postgresql.socket
Requires=postgresql.socket
|
Restart Policies#
1
2
3
4
5
6
7
8
9
10
11
12
| [Service]
# Restart behavior
Restart=always # Always restart
Restart=on-failure # Only on non-zero exit
Restart=on-abnormal # On signal, timeout, watchdog
Restart=on-abort # Only on signal
Restart=no # Never restart
# Timing
RestartSec=5 # Wait 5 seconds before restart
StartLimitIntervalSec=60
StartLimitBurst=3 # Max 3 restarts per 60 seconds
|
Restart Backoff#
1
2
3
4
5
| [Service]
Restart=always
RestartSec=1
RestartSteps=5
RestartMaxDelaySec=60
|
Restarts with exponential backoff: 1s, 2s, 4s, 8s, up to 60s max.
Environment and Working Directory#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [Service]
# Working directory
WorkingDirectory=/opt/myapp
# Single variables
Environment="NODE_ENV=production"
Environment="PORT=3000"
# From file
EnvironmentFile=/etc/myapp/env
EnvironmentFile=-/etc/myapp/env.local # - means optional
# User/group
User=myapp
Group=myapp
|
Process Management#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| [Service]
# Timeouts
TimeoutStartSec=30
TimeoutStopSec=30
# Stop signal
KillSignal=SIGTERM
KillMode=mixed # SIGTERM to main, SIGKILL to remaining
# Graceful shutdown commands
ExecStop=/usr/bin/myapp-stop
ExecStopPost=/usr/bin/cleanup.sh
# Pre/post commands
ExecStartPre=/usr/bin/check-deps.sh
ExecStartPost=/usr/bin/notify-started.sh
|
KillMode Options#
- control-group (default) - Kill all processes in cgroup
- mixed - SIGTERM to main, SIGKILL to others
- process - Kill only main process
- none - Don’t kill anything
Security Hardening#
Basic Hardening#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [Service]
# Run as non-root
User=myapp
Group=myapp
# Filesystem restrictions
ProtectSystem=strict # /usr, /boot, /efi read-only
ProtectHome=yes # /home, /root, /run/user inaccessible
PrivateTmp=yes # Private /tmp and /var/tmp
ReadWritePaths=/var/lib/myapp
# Capabilities
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
|
Network Isolation#
1
2
3
4
5
6
7
8
| [Service]
# Restrict network
PrivateNetwork=yes # No network access
RestrictAddressFamilies=AF_INET AF_INET6 # Limit to IP
# Restrict system calls
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
|
Full Lockdown Example#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| [Service]
User=myapp
Group=myapp
# Filesystem
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ReadWritePaths=/var/lib/myapp
ReadOnlyPaths=/etc/myapp
# Capabilities
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
# System calls
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged
# Other
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
|
Logging#
1
2
3
4
5
6
7
8
9
10
11
| [Service]
# Standard output/error to journal
StandardOutput=journal
StandardError=journal
# Or to file
StandardOutput=append:/var/log/myapp/out.log
StandardError=append:/var/log/myapp/err.log
# Syslog identifier (for journal filtering)
SyslogIdentifier=myapp
|
View logs:
1
2
3
| journalctl -u myapp.service
journalctl -u myapp.service -f # Follow
journalctl -u myapp.service --since "1 hour ago"
|
Resource Limits#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| [Service]
# Memory
MemoryMax=2G
MemoryHigh=1.5G # Soft limit, throttled above
# CPU
CPUQuota=200% # 2 CPU cores max
CPUWeight=50 # Scheduling priority (1-10000)
# IO
IOWeight=50
IOReadBandwidthMax=/dev/sda 100M
# Processes
TasksMax=100
# File descriptors
LimitNOFILE=65535
|
Socket Activation#
Start service on-demand when socket receives connection:
1
2
3
4
5
6
7
8
9
10
| # /etc/systemd/system/myapp.socket
[Unit]
Description=My App Socket
[Socket]
ListenStream=8080
Accept=no
[Install]
WantedBy=sockets.target
|
1
2
3
4
5
6
7
8
| # /etc/systemd/system/myapp.service
[Unit]
Description=My App
Requires=myapp.socket
[Service]
Type=simple
ExecStart=/usr/bin/myapp
|
1
2
3
| systemctl enable myapp.socket
systemctl start myapp.socket
# Service starts when connection arrives on :8080
|
Template Units#
For multiple instances:
1
2
3
4
5
6
7
| # /etc/systemd/system/myapp@.service
[Unit]
Description=My App instance %i
[Service]
ExecStart=/usr/bin/myapp --instance=%i
EnvironmentFile=/etc/myapp/%i.env
|
1
2
3
| systemctl start myapp@worker1
systemctl start myapp@worker2
systemctl status myapp@worker1
|
Complete Production Example#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
| # /etc/systemd/system/webapp.service
[Unit]
Description=Production Web Application
Documentation=https://docs.example.com/webapp
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service
Wants=redis.service
[Service]
Type=notify
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
EnvironmentFile=/etc/webapp/env
EnvironmentFile=-/etc/webapp/env.local
ExecStartPre=/opt/webapp/bin/check-deps
ExecStart=/opt/webapp/bin/server
ExecReload=/bin/kill -HUP $MAINPID
ExecStopPost=/opt/webapp/bin/cleanup
Restart=always
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3
TimeoutStartSec=30
TimeoutStopSec=30
WatchdogSec=30
# Security
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
ReadWritePaths=/var/lib/webapp /var/log/webapp
# Resources
MemoryMax=4G
CPUQuota=400%
TasksMax=512
LimitNOFILE=65535
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=webapp
[Install]
WantedBy=multi-user.target
|
Quick Reference#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| # Reload after editing service files
systemctl daemon-reload
# Enable/start
systemctl enable myapp
systemctl start myapp
# Status and logs
systemctl status myapp
journalctl -u myapp -f
# Analyze security
systemd-analyze security myapp
# Check service file syntax
systemd-analyze verify myapp.service
# Show all settings
systemctl show myapp
|
Good service files prevent 3 AM pages. Add proper dependencies so services start in order. Set restart policies so transient failures recover automatically. Apply security hardening so compromises are contained. The extra 10 minutes writing a proper service file saves hours of debugging later.