Devops

When GitHub-hosted runners aren’t enough—when you need GPU access, specific hardware, private network connectivity, or just want to stop paying per-minute—self-hosted runners are the answer. Why Self-Hosted? Performance: Your hardware, your speed. No cold starts, local caching, faster artifact access. Cost: After a certain threshold, self-hosted is dramatically cheaper. GitHub-hosted minutes add up fast for active repos. Access: Private networks, internal services, specialized hardware, air-gapped environments. Control: Exact OS versions, pre-installed dependencies, custom security configurations. ...

rsync synchronizes files between locations — local to local, local to remote, remote to local. It’s smart: it only transfers what’s changed, making it fast for incremental backups and deployments. Basic Syntax 1 rsync [options] source destination Local Sync 1 2 3 4 5 6 7 8 9 # Copy directory rsync -av /source/dir/ /dest/dir/ # Copy directory (trailing slash matters!) rsync -av /source/dir /dest/ # Creates /dest/dir/ rsync -av /source/dir/ /dest/ # Contents into /dest/ # Dry run (show what would happen) rsync -avn /source/ /dest/ Remote Sync (SSH) 1 2 3 4 5 6 7 8 9 10 11 # Local to remote rsync -av /local/dir/ user@remote:/remote/dir/ # Remote to local rsync -av user@remote:/remote/dir/ /local/dir/ # Different SSH port rsync -av -e 'ssh -p 2222' /local/ user@remote:/remote/ # With SSH key rsync -av -e 'ssh -i ~/.ssh/mykey' /local/ user@remote:/remote/ Common Options 1 2 3 4 5 6 7 8 9 -a, --archive Archive mode (preserves permissions, timestamps, etc.) -v, --verbose Verbose output -n, --dry-run Show what would be transferred -z, --compress Compress during transfer -P Progress + partial (resume interrupted transfers) --progress Show progress --delete Delete files in dest not in source -r, --recursive Recurse into directories -h, --human-readable Human-readable sizes Archive Mode (-a) -a is equivalent to -rlptgoD: ...

Cron has been scheduling tasks on Unix systems since 1975. It’s simple, reliable, and available everywhere. But that simplicity hides gotchas that break jobs in production. Cron Syntax ┌ │ │ │ │ │ ─ ─ ┌ │ │ │ │ ─ ─ ─ ─ ┌ │ │ │ ─ ─ ─ ─ ─ ─ ┌ │ │ ─ ─ ─ ─ ─ ─ ─ ─ ┌ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ c ─ ─ ─ ─ ─ o ─ ─ ─ ─ ─ m ─ ─ ─ ─ ─ m ─ ─ ─ ─ a m ─ ─ ─ ─ n i ─ ─ ─ d n h ─ ─ ─ u o ─ ─ t u d ─ ─ e r a ─ y m ─ ( ( o 0 0 o n d - - f t a 5 2 h y 9 3 m ) ) o ( o n 1 f t - h 1 w 2 e ( ) e 1 k - 3 ( 1 0 ) - 7 , 0 a n d 7 a r e S u n d a y ) Common Schedules 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Every minute * * * * * /path/to/script.sh # Every hour at minute 0 0 * * * * /path/to/script.sh # Every day at midnight 0 0 * * * /path/to/script.sh # Every day at 2:30 AM 30 2 * * * /path/to/script.sh # Every Monday at 9 AM 0 9 * * 1 /path/to/script.sh # Every 15 minutes */15 * * * * /path/to/script.sh # Every weekday at 6 PM 0 18 * * 1-5 /path/to/script.sh # First day of every month at midnight 0 0 1 * * /path/to/script.sh Special Strings 1 2 3 4 5 6 7 8 @reboot # Run once at startup @yearly # 0 0 1 1 * @annually # Same as @yearly @monthly # 0 0 1 * * @weekly # 0 0 * * 0 @daily # 0 0 * * * @midnight # Same as @daily @hourly # 0 * * * * Editing Crontabs 1 2 3 4 5 6 7 8 9 10 11 # Edit current user's crontab crontab -e # List current user's crontab crontab -l # Edit another user's crontab (as root) crontab -u username -e # Remove all cron jobs (careful!) crontab -r System Crontabs 1 2 3 4 5 6 7 8 9 10 11 # System-wide crontab (includes user field) /etc/crontab # Drop-in directory (no user field needed) /etc/cron.d/ # Periodic directories (scripts run by run-parts) /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /etc/crontab format includes username: ...

If you’re still typing ssh -i ~/.ssh/prod-key.pem -p 2222 ubuntu@ec2-54-123-45-67.compute-1.amazonaws.com, you’re working too hard. SSH config files let you define aliases with all your connection details. Basic Config Create or edit ~/.ssh/config: H o s t H U P I p o s o d r s e r e o t r t n d N t a u 2 i m b 2 t e u 2 y n 2 F e t i c u l 2 e - 5 ~ 4 / - . 1 s 2 s 3 h - / 4 p 5 r - o 6 d 7 - . k c e o y m . p p u e t m e - 1 . a m a z o n a w s . c o m Now connect with just: ...

Two commands solve 90% of search problems on Unix systems: grep for text patterns and find for file locations. Master these and you’ll navigate any codebase. grep Basics 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Search for pattern in file grep "error" logfile.txt # Case insensitive grep -i "error" logfile.txt # Show line numbers grep -n "error" logfile.txt # Count matches grep -c "error" logfile.txt # Invert match (lines NOT matching) grep -v "debug" logfile.txt grep in Multiple Files 1 2 3 4 5 6 7 8 9 10 11 # Search all files in directory grep "TODO" *.py # Recursive search grep -r "TODO" src/ # Show only filenames grep -l "TODO" *.py # Show filenames with no match grep -L "TODO" *.py grep with Context 1 2 3 4 5 6 7 8 # 3 lines before match grep -B 3 "error" logfile.txt # 3 lines after match grep -A 3 "error" logfile.txt # 3 lines before and after grep -C 3 "error" logfile.txt grep Regular Expressions 1 2 3 4 5 6 7 8 9 10 11 # Basic regex (default) grep "error.*failed" logfile.txt # Extended regex grep -E "error|warning|critical" logfile.txt # Or use egrep egrep "error|warning" logfile.txt # Perl regex (most powerful) grep -P "\d{4}-\d{2}-\d{2}" logfile.txt Common Patterns 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # IP addresses grep -E "\b[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\b" access.log # Email addresses grep -E "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" users.txt # URLs grep -E "https?://[^\s]+" document.txt # Timestamps grep -P "\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}" app.log # Word boundaries grep -w "error" logfile.txt # Won't match "errors" or "error_code" grep with Exclusions 1 2 3 4 5 6 7 8 # Exclude directories grep -r "TODO" --exclude-dir=node_modules --exclude-dir=.git . # Exclude file patterns grep -r "TODO" --exclude="*.min.js" --exclude="*.map" . # Include only certain files grep -r "TODO" --include="*.py" --include="*.js" . find Basics 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Find by name find . -name "*.py" # Case insensitive name find . -iname "readme*" # Find directories find . -type d -name "test*" # Find files find . -type f -name "*.log" # Find links find . -type l find by Time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Modified in last 7 days find . -mtime -7 # Modified more than 30 days ago find . -mtime +30 # Modified in last 60 minutes find . -mmin -60 # Accessed in last day find . -atime -1 # Changed (metadata) in last day find . -ctime -1 find by Size 1 2 3 4 5 6 7 8 9 10 11 # Files larger than 100MB find . -size +100M # Files smaller than 1KB find . -size -1k # Files exactly 0 bytes (empty) find . -size 0 # Size units: c (bytes), k (KB), M (MB), G (GB) find . -size +1G find by Permissions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Executable files find . -perm /u+x -type f # World-writable files find . -perm -002 # SUID files find . -perm -4000 # Files owned by user find . -user root # Files owned by group find . -group www-data find with Actions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Print (default) find . -name "*.log" -print # Delete (careful!) find . -name "*.tmp" -delete # Execute command for each file find . -name "*.py" -exec wc -l {} \; # Execute command with all files at once find . -name "*.py" -exec wc -l {} + # Interactive delete find . -name "*.bak" -ok rm {} \; find Logical Operators 1 2 3 4 5 6 7 8 9 10 11 # AND (implicit) find . -name "*.py" -size +100k # OR find . -name "*.py" -o -name "*.js" # NOT find . ! -name "*.pyc" # Grouping find . \( -name "*.py" -o -name "*.js" \) -size +10k Combining grep and find 1 2 3 4 5 6 7 8 9 10 11 # Find files and grep in them find . -name "*.py" -exec grep -l "import os" {} \; # More efficient with xargs find . -name "*.py" | xargs grep -l "import os" # Handle spaces in filenames find . -name "*.py" -print0 | xargs -0 grep -l "import os" # Find recently modified files with pattern find . -name "*.log" -mtime -1 -exec grep "ERROR" {} + Practical Examples Find Large Files 1 2 3 4 5 # Top 10 largest files find . -type f -exec du -h {} + | sort -rh | head -10 # Files over 100MB, sorted find . -type f -size +100M -exec ls -lh {} \; | sort -k5 -h Find and Clean 1 2 3 4 5 6 7 8 9 # Remove old log files find /var/log -name "*.log" -mtime +30 -delete # Remove empty directories find . -type d -empty -delete # Remove Python cache find . -type d -name "__pycache__" -exec rm -rf {} + find . -name "*.pyc" -delete Search Code 1 2 3 4 5 6 7 8 # Find function definitions grep -rn "def " --include="*.py" src/ # Find TODO comments grep -rn "TODO\|FIXME\|XXX" --include="*.py" . # Find imports grep -r "^import\|^from" --include="*.py" src/ | sort -u Search Logs 1 2 3 4 5 6 7 8 # Errors in last hour find /var/log -name "*.log" -mmin -60 -exec grep -l "ERROR" {} \; # Count errors per file find /var/log -name "*.log" -exec sh -c 'echo "$1: $(grep -c ERROR "$1")"' _ {} \; # Unique IPs from access log grep -oE "\b[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\b" access.log | sort -u Find Duplicates 1 2 3 4 5 # Find files with same name find . -type f -name "*.py" | xargs -I{} basename {} | sort | uniq -d # Find by content hash (requires md5sum) find . -type f -exec md5sum {} \; | sort | uniq -w32 -d Performance Tips 1 2 3 4 5 6 7 8 9 10 11 # Stop at first match (faster) grep -m 1 "pattern" largefile.txt # Use fixed strings when possible (faster than regex) grep -F "exact string" file.txt # Limit find depth find . -maxdepth 2 -name "*.py" # Prune directories find . -path ./node_modules -prune -o -name "*.js" -print ripgrep (Modern Alternative) If available, rg is faster than grep: ...

Systemd replaced init scripts on most Linux distributions. Instead of shell scripts with start/stop logic, you write declarative unit files that tell systemd what to run and how. Basic Service File 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # /etc/systemd/system/myapp.service [Unit] Description=My Application After=network.target [Service] Type=simple User=myapp WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/server Restart=always [Install] WantedBy=multi-user.target Enable and start: ...

YAML is everywhere — Kubernetes, Docker Compose, Ansible, GitHub Actions, CI/CD pipelines. It looks friendly until you spend an hour debugging why on became true or your port number turned into octal. Here are the traps and how to avoid them. The Norway Problem 1 2 3 4 5 # What you wrote country: NO # What YAML parsed country: false YAML interprets NO, no, No, OFF, off, Off as boolean false. Same with YES, yes, Yes, ON, on, On as true. ...

Git is flexible enough to support almost any workflow. That flexibility is both a blessing and a curse — without clear patterns, teams end up with merge conflicts, lost work, and frustration. Here are workflows that work. Solo Developer: Simple Trunk When you’re working alone, keep it simple: 1 2 3 4 5 6 7 8 # Work directly on main git checkout main git pull # Make changes git add . git commit -m "Add feature X" git push That’s it. No branches, no PRs, no ceremony. Save complexity for when you need it. ...

Bash scripts start simple and grow complex. A quick automation becomes critical infrastructure. Here’s how to write scripts that don’t become maintenance nightmares. Start Every Script Right 1 2 3 #!/usr/bin/env bash set -euo pipefail IFS=$'\n\t' What these do: #!/usr/bin/env bash — Portable shebang, finds bash in PATH set -e — Exit on any error set -u — Error on undefined variables set -o pipefail — Pipeline fails if any command fails IFS=$'\n\t' — Safer word splitting (no spaces) The Debugging Version 1 2 #!/usr/bin/env bash set -euxo pipefail The -x prints each command before execution. Remove for production. ...

The twelve-factor app methodology made environment variables the standard for configuration. They’re simple, universal, and keep secrets out of code. But there are right and wrong ways to use them. The Basics 1 2 3 4 5 6 7 8 9 10 11 # Set in current shell export DATABASE_URL="postgres://localhost/mydb" # Set for single command DATABASE_URL="postgres://localhost/mydb" ./myapp # Check if set echo $DATABASE_URL # Unset unset DATABASE_URL .env Files Don’t commit secrets. Use .env files for local development: ...