Devops

Most systemd services run with full system access by default. That’s fine until one gets compromised. Systemd provides powerful sandboxing options that most people never use. The Basics: User and Group Never run services as root if they don’t need it: 1 2 3 [Service] User=myapp Group=myapp Create a dedicated user: 1 sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp Filesystem Restrictions Read-Only Root Make the entire filesystem read-only: ...

Something’s broken. It worked last week. Somewhere in the 47 commits since then, someone introduced a bug. You could check each commit manually, or you could let git do the work. git bisect performs a binary search through your commit history to find the exact commit that introduced a problem. Instead of checking 47 commits, you check about 6. The Basic Workflow 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Start bisecting git bisect start # Mark current commit as bad (has the bug) git bisect bad # Mark a known good commit (before the bug existed) git bisect good v1.2.0 # or git bisect good abc123 # Git checks out a commit halfway between good and bad # Test it, then tell git the result: git bisect good # This commit doesn't have the bug # or git bisect bad # This commit has the bug # Git narrows down and checks out another commit # Repeat until git finds the first bad commit # When done, return to original state git bisect reset Example Session 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 $ git bisect start $ git bisect bad # HEAD is broken $ git bisect good v2.0.0 # v2.0.0 worked fine Bisecting: 23 revisions left to test after this (roughly 5 steps) [abc123...] Add caching layer $ ./run-tests.sh Tests pass! $ git bisect good Bisecting: 11 revisions left to test after this (roughly 4 steps) [def456...] Refactor auth module $ ./run-tests.sh FAIL! $ git bisect bad Bisecting: 5 revisions left to test after this (roughly 3 steps) [ghi789...] Update dependencies # ... continue until: abc123def456789 is the first bad commit commit abc123def456789 Author: Someone <someone@example.com> Date: Mon Feb 20 14:30:00 2024 Fix edge case in login flow This commit introduced the bug! $ git bisect reset Automated Bisecting If you have a script that can test for the bug: ...

Nginx sits in front of most web applications. It handles SSL, load balancing, static files, and proxying — all while being incredibly efficient. Here are the configurations you’ll actually use. Basic Structure 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 # /etc/nginx/nginx.conf user www-data; worker_processes auto; pid /run/nginx.pid; events { worker_connections 1024; multi_accept on; } http { # Basic settings sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; server_tokens off; # Hide nginx version # MIME types include /etc/nginx/mime.types; default_type application/octet-stream; # Logging access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; # Gzip gzip on; gzip_vary on; gzip_min_length 1024; gzip_types text/plain text/css application/json application/javascript text/xml application/xml; # Virtual hosts include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } Simple Reverse Proxy 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # /etc/nginx/sites-available/myapp server { listen 80; server_name myapp.example.com; location / { proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } SSL with Let’s Encrypt After running certbot --nginx: ...

Make has been around since 1976. It’s installed on virtually every Unix system. And while it was designed for compiling C programs, it’s become a universal task runner for any project. No npm, no pip, no cargo — just make. Why Make? Zero dependencies — Already on your system Declarative — Describe what you want, not how to get it Incremental — Only runs what’s needed Self-documenting — make help shows available targets Universal — Works the same on Linux, macOS, CI systems Basic Structure 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # Variables APP_NAME := myapp VERSION := 1.0.0 # Default target (runs when you just type 'make') .DEFAULT_GOAL := help # Phony targets don't create files .PHONY: build test clean help build: go build -o $(APP_NAME) ./cmd/$(APP_NAME) test: go test ./... clean: rm -f $(APP_NAME) help: @echo "Available targets:" @echo " build - Build the application" @echo " test - Run tests" @echo " clean - Remove build artifacts" Self-Documenting Makefiles The best pattern: automatic help generation from comments. ...

Terraform state is both essential and dangerous. It’s how Terraform knows what exists, what changed, and what to do. Mismanage it, and you’ll either destroy production or spend hours untangling drift. What State Actually Is State is Terraform’s record of reality. It maps your configuration to real resources: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 { "resources": [ { "type": "aws_instance", "name": "web", "instances": [ { "attributes": { "id": "i-0abc123def456", "ami": "ami-12345678", "instance_type": "t3.medium" } } ] } ] } Without state, Terraform would: ...

SSH is the workhorse of remote access. But most people only scratch the surface. Here’s how to use it like a pro. SSH Config: Stop Typing So Much Instead of: 1 ssh -i ~/.ssh/prod-key.pem -p 2222 ubuntu@ec2-54-123-45-67.compute-1.amazonaws.com Create ~/.ssh/config: H H H o o o s s s t t t H U P I H U I F P U p o s o d s o s d o r s r s e r e t s e e r . o e o t r t n a t r n w i x r d N t g N t a n y a u 2 i i a d i r t J a m b 2 t n m e t d e u d e u 2 y g e p y A r m m n 2 F l F g n p i e t i s o i e a n c u l t y l n l b 2 e a e t a - g s 5 ~ i ~ y t 4 / n / e i - . g . s o 1 s . s n 2 s e s 3 h x h - / a / 4 p m s 5 r p t - o l a 6 d e g 7 - . i . k c n c e o g o y m - m . k p p e u e y t m . e p - e 1 m . a m a z o n a w s . c o m Now just: ...

Ansible playbooks start simple and grow complex. A quick server setup becomes infrastructure-as-code for dozens of machines. Here are patterns that keep playbooks maintainable as they scale. Project Structure a ├ ├ │ │ │ │ │ │ │ │ ├ │ │ │ ├ │ │ │ └ n ─ ─ ─ ─ ─ s ─ ─ ─ ─ ─ i b a i ├ │ │ │ │ └ p ├ ├ └ r ├ ├ └ f l n n ─ ─ l ─ ─ ─ o ─ ─ ─ i e s v ─ ─ a ─ ─ ─ l ─ ─ ─ l / i e y e e b n p ├ └ s ├ └ b s w d s c n p s l t r ─ ─ t ─ ─ o i e a / o g o / e o o ─ ─ a ─ ─ o t b t m i s . r d g k e s a m n t c y u h g ├ └ i h g s . e b o x g f / c o r ─ ─ n o r y r a n r g t s o ─ ─ g s m v s e i t u / t u l e e s o s p a w s p r s / n . _ l e . _ s . y v l b y v . y m a . s m a y m l r y e l r m l s m r s l / l v / e r s . y m l ansible.cfg 1 2 3 4 5 6 7 8 9 [defaults] inventory = inventory/production roles_path = roles host_key_checking = False retry_files_enabled = False [ssh_connection] pipelining = True control_path = /tmp/ansible-%%r@%%h:%%p Inventory Patterns YAML Inventory (Preferred) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # inventory/production/hosts.yml all: children: webservers: hosts: web1.example.com: web2.example.com: vars: http_port: 80 databases: hosts: db1.example.com: postgres_version: 15 db2.example.com: postgres_version: 14 loadbalancers: hosts: lb1.example.com: Group Variables 1 2 3 4 5 6 7 8 9 10 11 12 13 # inventory/production/group_vars/all.yml --- ansible_user: deploy ansible_python_interpreter: /usr/bin/python3 ntp_servers: - 0.pool.ntp.org - 1.pool.ntp.org # inventory/production/group_vars/webservers.yml --- nginx_worker_processes: auto nginx_worker_connections: 1024 app_root: /var/www/app Role Structure r ├ │ ├ │ ├ │ ├ │ ├ │ ├ │ └ o ─ ─ ─ ─ ─ ─ ─ l ─ ─ ─ ─ ─ ─ ─ e s d └ v └ t └ h └ t └ f └ m └ / e ─ a ─ a ─ a ─ e ─ i ─ e ─ n f ─ r ─ s ─ n ─ m ─ l ─ t ─ g a s k d p e a i u m / m s m l m l n s s / m n l a a / a e a a g / s a x t i i i r i t i l i / s n n n s n e n - n / . . . / . s x p . y y y y / . a y m m m m c r m l l l l o a l n m f s . . j c 2 o n # # # # # f # D R T H J R e o a a i o f l s n n l a e k d j e u l a l v e e 2 m t a n r e r t s t t v i r e a a a y ( m d r b r p a i l p e l t a e o s a a b s i t t l n a e a e ( t r s n s h t d i ( g s d l h e e o r p w p v e e r i n s i c d t o e e r s n p i , c r t i i y e e o ) t s r c i . t ) y ) defaults/main.yml 1 2 3 4 5 6 --- # Overridable defaults nginx_worker_processes: auto nginx_worker_connections: 768 nginx_keepalive_timeout: 65 nginx_server_tokens: "off" tasks/main.yml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 --- - name: Install nginx ansible.builtin.apt: name: nginx state: present update_cache: true become: true - name: Configure nginx ansible.builtin.template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf owner: root group: root mode: '0644' become: true notify: Reload nginx - name: Enable and start nginx ansible.builtin.systemd: name: nginx enabled: true state: started become: true handlers/main.yml 1 2 3 4 5 6 7 8 9 10 11 12 --- - name: Reload nginx ansible.builtin.systemd: name: nginx state: reloaded become: true - name: Restart nginx ansible.builtin.systemd: name: nginx state: restarted become: true Task Patterns Idempotent Tasks 1 2 3 4 5 6 7 8 9 10 # Good - idempotent - name: Ensure user exists ansible.builtin.user: name: deploy state: present groups: [sudo, docker] # Avoid - not idempotent - name: Add user ansible.builtin.command: useradd deploy Conditional Execution 1 2 3 4 5 6 7 8 9 10 11 - name: Install package (Debian) ansible.builtin.apt: name: nginx state: present when: ansible_os_family == "Debian" - name: Install package (RedHat) ansible.builtin.dnf: name: nginx state: present when: ansible_os_family == "RedHat" Loops 1 2 3 4 5 6 7 8 9 10 - name: Create users ansible.builtin.user: name: "{{ item.name }}" groups: "{{ item.groups }}" state: present loop: - { name: alice, groups: [developers] } - { name: bob, groups: [developers, sudo] } loop_control: label: "{{ item.name }}" # Cleaner output Blocks for Error Handling 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 - name: Deploy application block: - name: Pull latest code ansible.builtin.git: repo: "{{ app_repo }}" dest: "{{ app_root }}" version: "{{ app_version }}" - name: Install dependencies ansible.builtin.pip: requirements: "{{ app_root }}/requirements.txt" virtualenv: "{{ app_root }}/venv" - name: Run migrations ansible.builtin.command: cmd: "{{ app_root }}/venv/bin/python manage.py migrate" chdir: "{{ app_root }}" rescue: - name: Notify on failure ansible.builtin.debug: msg: "Deployment failed, rolling back" - name: Rollback to previous version ansible.builtin.git: repo: "{{ app_repo }}" dest: "{{ app_root }}" version: "{{ previous_version }}" always: - name: Restart application ansible.builtin.systemd: name: myapp state: restarted Variable Precedence From lowest to highest priority: ...

“It’s always DNS” is a meme because it’s usually true. When something network-related breaks, DNS is the first suspect. Here’s how to investigate. The Essential Tools dig - The DNS Swiss Army Knife 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Basic lookup dig example.com # Specific record type dig example.com MX dig example.com TXT dig example.com CNAME # Short output (just the answer) dig +short example.com # Query a specific nameserver dig @8.8.8.8 example.com # Trace the full resolution path dig +trace example.com nslookup - Quick and Simple 1 2 3 4 5 6 7 8 9 10 # Basic lookup nslookup example.com # Query specific server nslookup example.com 8.8.8.8 # Interactive mode nslookup > set type=MX > example.com host - Even Simpler 1 2 3 4 5 # Basic lookup host example.com # Specific record type host -t MX example.com Common DNS Problems Problem: Domain Not Resolving Symptoms: NXDOMAIN or SERVFAIL errors ...

When you have one server, you SSH in and grep the logs. When you have fifty servers, that stops working. Log aggregation is how you make “what happened?” answerable at scale. The Pipeline Architecture Every log aggregation system follows the same basic pattern: ┌ │ └ ─ ─ ─ S ─ ─ o ─ ─ u ─ ─ r ─ │ │ └ ─ c ─ ─ ─ e ─ ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ─ ▶ ▶ ┌ │ └ ┌ │ └ ─ ─ ─ ─ ─ C ─ ─ ─ ─ o ─ ─ Q ─ ─ l ─ ─ u ─ ─ l ─ ─ e ─ ─ e ─ ─ r ─ ─ c ─ ─ y ─ ─ t ─ ─ ─ ─ ─ ─ ─ ┐ │ ┘ ┐ │ ┘ ─ ◀ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ P ─ ─ ─ r ─ ─ ─ o ─ ─ ─ c ─ ─ ─ e ─ ─ ─ s ─ ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ ─ ─ ─ S ─ ─ ─ t ─ ─ ─ o ─ │ ┘ ─ r ─ │ ─ e ─ ─ ─ ─ ─ ┐ │ ┘ Each stage has choices. Let’s walk through them. ...

Most production incidents I’ve debugged came down to configuration. A missing environment variable. A wrong database URL. A feature flag stuck in the wrong state. Code was fine; configuration was the problem. Configuration management is the unsexy work that prevents those 3 AM pages. The Core Principles 1. Separate Configuration from Code Configuration should never be baked into your application binary or container image. Wrong: 1 2 # Hardcoded in code DATABASE_URL = "postgres://prod:password@db.example.com/myapp" Also wrong: ...