Systemd

Your systemd service is “active (running)” but the application isn’t responding. No errors in systemctl status. The journal shows it started. Everything looks fine. Except it isn’t. This is one of the most frustrating debugging scenarios in Linux administration. Here’s how to actually figure out what’s wrong. The Problem: Green Status, Dead Application 1 2 3 4 5 $ systemctl status myapp ● myapp.service - My Application Loaded: loaded (/etc/systemd/system/myapp.service; enabled) Active: active (running) since Fri 2026-03-20 10:00:00 EDT; 2h ago Main PID: 12345 (myapp) Looks healthy. But curl localhost:8080 times out. What’s happening? ...

Systemd is the init system for most modern Linux distributions. Love it or hate it, you need to know it. Here’s how to manage services effectively. Basic Commands 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # Start/stop/restart sudo systemctl start nginx sudo systemctl stop nginx sudo systemctl restart nginx # Reload config without restart sudo systemctl reload nginx # Enable/disable at boot sudo systemctl enable nginx sudo systemctl disable nginx # Check status systemctl status nginx # List all services systemctl list-units --type=service # List failed services systemctl --failed Writing a Service Unit Create /etc/systemd/system/myapp.service: ...

Most systemd service files are written for functionality, not security. The defaults give services more access than they need—full filesystem visibility, network capabilities, and the ability to spawn processes anywhere. A few directives can dramatically reduce the blast radius if that service gets compromised. The Security Baseline Start with this template for any service that doesn’t need special privileges: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 [Service] # Run as dedicated user User=myservice Group=myservice # Filesystem restrictions ProtectSystem=strict ProtectHome=true PrivateTmp=true ReadWritePaths=/var/lib/myservice # Network restrictions (if service doesn't need network) # PrivateNetwork=true # Capability restrictions NoNewPrivileges=true CapabilityBoundingSet= AmbientCapabilities= # System call filtering SystemCallArchitectures=native SystemCallFilter=@system-service SystemCallFilter=~@privileged @resources # Additional hardening ProtectKernelTunables=true ProtectKernelModules=true ProtectKernelLogs=true ProtectControlGroups=true RestrictRealtime=true RestrictSUIDSGID=true MemoryDenyWriteExecute=true LockPersonality=true Understanding Each Directive Filesystem Protection 1 ProtectSystem=strict Mounts /usr, /boot, and /efi read-only. The strict level also makes /etc read-only. Use full if the service needs to write to /etc. ...

Cron works. It’s also from 1975. systemd timers offer logging integration, dependency handling, and more flexible scheduling. Here’s how to use them. Why Timers Over Cron? Logging: Output goes to journald automatically Dependencies: Wait for network, mounts, or other services Flexibility: Calendar events, monotonic timers, randomized delays Visibility: systemctl list-timers shows everything Consistency: Same management as other systemd units Basic Structure A timer needs two files: A .timer unit (the schedule) A .service unit (the job) Place them in /etc/systemd/system/ (system-wide) or ~/.config/systemd/user/ (user). ...

If you’re running services on Linux, you’re almost certainly using systemd. But there’s a gap between knowing systemctl start nginx and actually writing your own robust service units. Let’s close that gap. The Anatomy of a Service Unit A systemd service unit lives in /etc/systemd/system/ and has three main sections: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [Unit] Description=My Application Service After=network.target Wants=network-online.target [Service] Type=simple User=appuser Group=appgroup WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/server Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target Let’s break down what matters: ...

Systemd services look simple until they don’t start, restart unexpectedly, or fail silently. Here’s how to write service files that work reliably in production. Basic Structure 1 2 3 4 5 6 7 8 9 10 11 12 # /etc/systemd/system/myapp.service [Unit] Description=My Application After=network.target [Service] Type=simple ExecStart=/usr/bin/myapp Restart=always [Install] WantedBy=multi-user.target Three sections, three purposes: [Unit] - What is this, what does it depend on [Service] - How to run it [Install] - When to start it Service Types Type=simple (Default) 1 2 3 [Service] Type=simple ExecStart=/usr/bin/myapp Systemd considers the service started immediately when ExecStart runs. Use when your process stays in foreground. ...

Cron has run scheduled tasks since 1975. It works, but systemd timers offer significant advantages: integrated logging, dependency management, randomized delays, and calendar-based scheduling that actually makes sense. Why Switch from Cron? Logging: Timer output goes to journald. No more digging through mail or custom log files. Dependencies: Wait for network, mounts, or other services before running. Accuracy: Monotonic timers don’t drift. Calendar timers handle DST correctly. Visibility: systemctl list-timers shows all scheduled jobs and when they’ll run next. ...

systemd’s journal collects logs from all services, the kernel, and system messages in one place. journalctl is your tool for searching, filtering, and following those logs. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # Show all logs (oldest first) journalctl # Show all logs (newest first) journalctl -r # Follow new entries (like tail -f) journalctl -f # Show only errors and above journalctl -p err Filter by Time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # Since boot journalctl -b # Previous boot journalctl -b -1 # Since specific time journalctl --since "2024-02-25 10:00:00" # Until specific time journalctl --until "2024-02-25 12:00:00" # Time range journalctl --since "1 hour ago" journalctl --since "2024-02-25" --until "2024-02-26" # Relative times journalctl --since "yesterday" journalctl --since "10 minutes ago" Filter by Unit (Service) 1 2 3 4 5 6 7 8 9 10 11 # Specific service journalctl -u nginx # Multiple services journalctl -u nginx -u php-fpm # Follow specific service journalctl -u nginx -f # Service since boot journalctl -u nginx -b Filter by Priority Priority levels (0=emergency to 7=debug): ...

Systemd replaced init scripts on most Linux distributions. Instead of shell scripts with start/stop logic, you write declarative unit files that tell systemd what to run and how. Basic Service File 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # /etc/systemd/system/myapp.service [Unit] Description=My Application After=network.target [Service] Type=simple User=myapp WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/server Restart=always [Install] WantedBy=multi-user.target Enable and start: ...

Most systemd services run with full system access by default. That’s fine until one gets compromised. Systemd provides powerful sandboxing options that most people never use. The Basics: User and Group Never run services as root if they don’t need it: 1 2 3 [Service] User=myapp Group=myapp Create a dedicated user: 1 sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp Filesystem Restrictions Read-Only Root Make the entire filesystem read-only: ...