Devops

curl is the universal language of HTTP. Every API doc includes curl examples. Every debugging session starts with “can you curl it?” If you’re not comfortable with curl, you’re missing the most portable tool in your kit. Basic Requests 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # GET (default) curl https://api.example.com/users # With headers shown curl -i https://api.example.com/users # Headers only curl -I https://api.example.com/users # Silent (no progress bar) curl -s https://api.example.com/users # Follow redirects curl -L https://example.com/redirect # Verbose (debug mode) curl -v https://api.example.com/users HTTP Methods 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # POST curl -X POST https://api.example.com/users # PUT curl -X PUT https://api.example.com/users/1 # PATCH curl -X PATCH https://api.example.com/users/1 # DELETE curl -X DELETE https://api.example.com/users/1 # HEAD (headers only, like -I) curl -X HEAD https://api.example.com/users Sending Data Form Data 1 2 3 4 5 6 7 # URL-encoded form curl -X POST https://api.example.com/login \ -d "username=admin&password=secret" # From file curl -X POST https://api.example.com/login \ -d @credentials.txt JSON Data 1 2 3 4 5 6 7 8 9 10 11 12 # Inline JSON curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d '{"name":"Alice","email":"alice@example.com"}' # From file curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d @user.json # Using --json (curl 7.82+) curl --json '{"name":"Alice"}' https://api.example.com/users File Upload 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Single file curl -X POST https://api.example.com/upload \ -F "file=@document.pdf" # Multiple files curl -X POST https://api.example.com/upload \ -F "file1=@doc1.pdf" \ -F "file2=@doc2.pdf" # File with custom filename curl -X POST https://api.example.com/upload \ -F "file=@localname.pdf;filename=remote.pdf" # File with content type curl -X POST https://api.example.com/upload \ -F "file=@image.png;type=image/png" # Mixed form data and files curl -X POST https://api.example.com/upload \ -F "title=My Document" \ -F "file=@document.pdf" Headers 1 2 3 4 5 6 7 8 9 10 11 12 13 # Custom header curl -H "X-Custom-Header: value" https://api.example.com # Multiple headers curl -H "Accept: application/json" \ -H "X-API-Version: 2" \ https://api.example.com # User agent curl -A "MyApp/1.0" https://api.example.com # Referer curl -e "https://example.com" https://api.example.com Authentication Basic Auth 1 2 3 4 5 6 7 8 # Username and password curl -u username:password https://api.example.com # Prompt for password curl -u username https://api.example.com # In URL (not recommended) curl https://username:password@api.example.com Bearer Token 1 curl -H "Authorization: Bearer YOUR_TOKEN" https://api.example.com API Key 1 2 3 4 5 # In header curl -H "X-API-Key: YOUR_KEY" https://api.example.com # In query string curl "https://api.example.com?api_key=YOUR_KEY" OAuth 2.0 Flow 1 2 3 4 5 6 7 8 9 # Get access token curl -X POST https://auth.example.com/oauth/token \ -d "grant_type=client_credentials" \ -d "client_id=YOUR_CLIENT_ID" \ -d "client_secret=YOUR_SECRET" # Use token TOKEN="eyJ..." curl -H "Authorization: Bearer $TOKEN" https://api.example.com/resource Digest Auth 1 curl --digest -u username:password https://api.example.com Output Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Save to file curl -o output.html https://example.com # Save with remote filename curl -O https://example.com/file.zip # Save multiple files curl -O https://example.com/file1.zip -O https://example.com/file2.zip # Append to file curl https://example.com >> output.txt # Write headers to file curl -D headers.txt https://example.com # Output to stdout and file curl https://example.com | tee output.html Timeouts and Retries 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Connection timeout (seconds) curl --connect-timeout 5 https://api.example.com # Max time for entire operation curl -m 30 https://api.example.com # Retry on failure curl --retry 3 https://api.example.com # Retry with delay curl --retry 3 --retry-delay 5 https://api.example.com # Retry on specific errors curl --retry 3 --retry-all-errors https://api.example.com SSL/TLS 1 2 3 4 5 6 7 8 9 10 11 12 # Skip certificate verification (insecure!) curl -k https://self-signed.example.com # Use specific CA certificate curl --cacert /path/to/ca.crt https://api.example.com # Client certificate curl --cert client.crt --key client.key https://api.example.com # Force TLS version curl --tlsv1.2 https://api.example.com curl --tlsv1.3 https://api.example.com Proxy 1 2 3 4 5 6 7 8 9 10 11 # HTTP proxy curl -x http://proxy:8080 https://api.example.com # SOCKS5 proxy curl --socks5 localhost:1080 https://api.example.com # Proxy with auth curl -x http://user:pass@proxy:8080 https://api.example.com # No proxy for specific hosts curl --noproxy "localhost,*.internal" https://api.example.com Cookies 1 2 3 4 5 6 7 8 9 10 11 # Send cookie curl -b "session=abc123" https://api.example.com # Send cookies from file curl -b cookies.txt https://api.example.com # Save cookies to file curl -c cookies.txt https://api.example.com/login # Full session (save and send) curl -b cookies.txt -c cookies.txt https://api.example.com Response Inspection 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # HTTP status code only curl -s -o /dev/null -w "%{http_code}" https://api.example.com # Response time curl -s -o /dev/null -w "%{time_total}s" https://api.example.com # Detailed timing curl -s -o /dev/null -w " DNS: %{time_namelookup}s Connect: %{time_connect}s TLS: %{time_appconnect}s Start: %{time_starttransfer}s Total: %{time_total}s Size: %{size_download} bytes Speed: %{speed_download} bytes/sec " https://api.example.com # Content type curl -s -o /dev/null -w "%{content_type}" https://api.example.com Scripting Patterns Health Check 1 2 3 4 5 6 7 8 9 10 11 #!/bin/bash URL="https://api.example.com/health" STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$URL") if [ "$STATUS" -eq 200 ]; then echo "OK" exit 0 else echo "FAIL: HTTP $STATUS" exit 1 fi API Wrapper 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #!/bin/bash API_BASE="https://api.example.com" API_KEY="${API_KEY:?API_KEY required}" api_get() { curl -s -H "Authorization: Bearer $API_KEY" "$API_BASE$1" } api_post() { curl -s -X POST \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$2" \ "$API_BASE$1" } # Usage api_get "/users" | jq '.' api_post "/users" '{"name":"Alice"}' Retry with Backoff 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 #!/bin/bash URL="$1" MAX_RETRIES=5 RETRY_DELAY=1 for i in $(seq 1 $MAX_RETRIES); do RESPONSE=$(curl -s -w "\n%{http_code}" "$URL") STATUS=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') if [ "$STATUS" -eq 200 ]; then echo "$BODY" exit 0 fi echo "Attempt $i failed (HTTP $STATUS), retrying in ${RETRY_DELAY}s..." >&2 sleep $RETRY_DELAY RETRY_DELAY=$((RETRY_DELAY * 2)) done echo "Failed after $MAX_RETRIES attempts" >&2 exit 1 Parallel Requests 1 2 3 4 5 # Using xargs cat urls.txt | xargs -P 10 -I {} curl -s -o /dev/null -w "{}: %{http_code}\n" {} # Using GNU parallel parallel -j 10 curl -s -o /dev/null -w "{}: %{http_code}\n" ::: $(cat urls.txt) Config Files Create ~/.curlrc for defaults: ...

Every API returns JSON. Every config file is JSON. If you’re not fluent in jq, you’re copying data by hand like it’s 1995. The Basics 1 2 3 4 5 6 7 8 9 10 # Pretty print echo '{"name":"test","value":42}' | jq '.' # Extract a field echo '{"name":"test","value":42}' | jq '.name' # "test" # Raw output (no quotes) echo '{"name":"test","value":42}' | jq -r '.name' # test Working with APIs 1 2 3 4 5 6 7 8 # GitHub API curl -s https://api.github.com/users/torvalds | jq '.login, .public_repos' # Extract specific fields curl -s https://api.github.com/repos/stedolan/jq | jq '{name, stars: .stargazers_count, language}' # AWS CLI (already outputs JSON) aws ec2 describe-instances | jq '.Reservations[].Instances[] | {id: .InstanceId, state: .State.Name}' Array Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # Sample data DATA='[{"name":"alice","age":30},{"name":"bob","age":25},{"name":"carol","age":35}]' # First element echo $DATA | jq '.[0]' # Last element echo $DATA | jq '.[-1]' # Slice echo $DATA | jq '.[0:2]' # All names echo $DATA | jq '.[].name' # Array of names echo $DATA | jq '[.[].name]' # Length echo $DATA | jq 'length' Filtering 1 2 3 4 5 6 7 8 9 10 11 # Select by condition echo $DATA | jq '.[] | select(.age > 28)' # Multiple conditions echo $DATA | jq '.[] | select(.age > 25 and .name != "carol")' # Contains echo '[{"tags":["web","api"]},{"tags":["cli"]}]' | jq '.[] | select(.tags | contains(["api"]))' # Has key echo '{"a":1,"b":null}' | jq 'has("a"), has("c")' Transformation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Add/modify fields echo '{"name":"test"}' | jq '. + {status: "active", count: 0}' # Update existing field echo '{"count":5}' | jq '.count += 1' # Delete field echo '{"a":1,"b":2,"c":3}' | jq 'del(.b)' # Rename key echo '{"old_name":"value"}' | jq '{new_name: .old_name}' # Map over array echo '[1,2,3,4,5]' | jq 'map(. * 2)' # Map with objects echo $DATA | jq 'map({username: .name, birth_year: (2026 - .age)})' String Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Concatenation echo '{"first":"John","last":"Doe"}' | jq '.first + " " + .last' # String interpolation echo '{"name":"test","ver":"1.0"}' | jq '"\(.name)-\(.ver).tar.gz"' # Split echo '{"path":"/usr/local/bin"}' | jq '.path | split("/")' # Join echo '["a","b","c"]' | jq 'join(",")' # Upper/lower echo '"Hello World"' | jq 'ascii_downcase' echo '"Hello World"' | jq 'ascii_upcase' # Test regex echo '{"email":"test@example.com"}' | jq '.email | test("@")' # Replace echo '"hello world"' | jq 'gsub("world"; "jq")' Conditionals 1 2 3 4 5 6 7 8 9 10 11 # If-then-else echo '{"status":200}' | jq 'if .status == 200 then "ok" else "error" end' # Alternative operator (default value) echo '{"a":1}' | jq '.b // "default"' # Null handling echo '{"a":null}' | jq '.a // "was null"' # Error handling echo '{}' | jq '.missing.nested // "not found"' Grouping and Aggregation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 LOGS='[ {"level":"error","msg":"failed"}, {"level":"info","msg":"started"}, {"level":"error","msg":"timeout"}, {"level":"info","msg":"completed"} ]' # Group by field echo $LOGS | jq 'group_by(.level)' # Count per group echo $LOGS | jq 'group_by(.level) | map({level: .[0].level, count: length})' # Unique values echo $LOGS | jq '[.[].level] | unique' # Sort echo $DATA | jq 'sort_by(.age)' # Reverse sort echo $DATA | jq 'sort_by(.age) | reverse' # Min/max echo '[5,2,8,1,9]' | jq 'min, max' # Sum echo '[1,2,3,4,5]' | jq 'add' # Average echo '[1,2,3,4,5]' | jq 'add / length' Constructing Output 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Build new object curl -s https://api.github.com/users/torvalds | jq '{ username: .login, repos: .public_repos, profile: .html_url }' # Build array echo '{"users":[{"name":"a"},{"name":"b"}]}' | jq '[.users[].name]' # Multiple outputs to array echo '{"a":1,"b":2}' | jq '[.a, .b, .a + .b]' # Key-value pairs echo '{"a":1,"b":2}' | jq 'to_entries' # [{"key":"a","value":1},{"key":"b","value":2}] # Back to object echo '[{"key":"a","value":1}]' | jq 'from_entries' # Transform keys echo '{"old_a":1,"old_b":2}' | jq 'with_entries(.key |= ltrimstr("old_"))' Real-World Examples Parse AWS Instance List 1 2 3 4 5 aws ec2 describe-instances | jq -r ' .Reservations[].Instances[] | [.InstanceId, .State.Name, (.Tags[]? | select(.Key=="Name") | .Value) // "unnamed"] | @tsv ' Filter Docker Containers 1 2 3 4 5 6 docker inspect $(docker ps -q) | jq '.[] | { name: .Name, image: .Config.Image, status: .State.Status, ip: .NetworkSettings.IPAddress }' Process Log Files 1 2 3 4 5 6 7 # Count errors by type cat app.log | jq -s 'group_by(.error_type) | map({type: .[0].error_type, count: length}) | sort_by(.count) | reverse' # Extract errors from last hour cat app.log | jq --arg cutoff "$(date -d '1 hour ago' -Iseconds)" ' select(.timestamp > $cutoff and .level == "error") ' Transform Config Files 1 2 3 4 5 6 7 8 # Merge configs jq -s '.[0] * .[1]' base.json override.json # Update nested value jq '.database.host = "newhost.example.com"' config.json # Add to array jq '.allowed_ips += ["10.0.0.5"]' config.json Generate Reports 1 2 3 4 5 6 # Kubernetes pod status kubectl get pods -o json | jq -r ' .items[] | [.metadata.name, .status.phase, (.status.containerStatuses[0].restartCount // 0)] | @tsv ' | column -t Useful Flags 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 # Compact output (no pretty print) jq -c '.' # Raw output (no quotes on strings) jq -r '.name' # Raw input (treat input as string, not JSON) jq -R 'split(",")' # Slurp (read all inputs into array) cat *.json | jq -s '.' # Pass variable jq --arg name "test" '.name = $name' # Pass JSON variable jq --argjson count 42 '.count = $count' # Read from file jq --slurpfile users users.json '.users = $users' # Exit with error if output is null/false jq -e '.important_field' && echo "exists" # Sort keys in output jq -S '.' Output Formats 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Tab-separated echo $DATA | jq -r '.[] | [.name, .age] | @tsv' # CSV echo $DATA | jq -r '.[] | [.name, .age] | @csv' # URI encoding echo '{"q":"hello world"}' | jq -r '.q | @uri' # Base64 echo '{"data":"secret"}' | jq -r '.data | @base64' # Shell-safe echo '{"cmd":"echo hello"}' | jq -r '.cmd | @sh' Debugging 1 2 3 4 5 6 7 8 9 10 11 # Show type echo '{"a":[1,2,3]}' | jq '.a | type' # Show keys echo '{"a":1,"b":2}' | jq 'keys' # Debug output (shows intermediate values) echo '{"x":{"y":{"z":1}}}' | jq '.x | debug | .y | debug | .z' # Path to value echo '{"a":{"b":{"c":1}}}' | jq 'path(.. | select(. == 1))' Quick Reference 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 # Identity . # Field access .field .field.nested # Array access .[0] .[-1] .[2:5] # Iterate array .[] # Pipe .[] | .name # Collect into array [.[] | .name] # Object construction {newkey: .oldkey} # Conditionals if COND then A else B end VALUE // DEFAULT # Comparison ==, !=, <, >, <=, >= and, or, not # Array functions map(f), select(f), sort_by(f), group_by(f), unique, length, first, last, nth(n), flatten, reverse, contains(x), inside(x), add, min, max # String functions split(s), join(s), test(re), match(re), gsub(re;s), ascii_downcase, ascii_upcase, ltrimstr(s), rtrimstr(s), startswith(s), endswith(s) # Object functions keys, values, has(k), in(o), to_entries, from_entries, with_entries(f) # Type functions type, isnumber, isstring, isnull, isboolean, isarray, isobject jq turns JSON from a data format into a query language. Once you internalize the pipe-and-filter model, you’ll wonder how you ever survived without it. ...

Cron has run scheduled tasks since 1975. It works, but systemd timers offer significant advantages: integrated logging, dependency management, randomized delays, and calendar-based scheduling that actually makes sense. Why Switch from Cron? Logging: Timer output goes to journald. No more digging through mail or custom log files. Dependencies: Wait for network, mounts, or other services before running. Accuracy: Monotonic timers don’t drift. Calendar timers handle DST correctly. Visibility: systemctl list-timers shows all scheduled jobs and when they’ll run next. ...

The third factor of the 12-Factor App methodology states: “Store config in the environment.” Simple advice that’s surprisingly easy to get wrong. The Core Principle Configuration that varies between environments (dev, staging, production) should come from environment variables, not code. This includes: Database connection strings API keys and secrets Feature flags Service URLs Port numbers Log levels What stays in code: application logic, default behaviors, anything that doesn’t change between deploys. ...

Make was designed for compiling C programs in 1976. Nearly 50 years later, it’s still one of the most practical automation tools available—not for its original purpose, but as a universal task runner. Why Make in 2026? It’s already installed. Every Unix system has make. No npm install, no pip, no version managers. It’s declarative. Define what you want, not how to get there (with dependencies handled automatically). It’s documented. make help can list all your targets. The Makefile itself is documentation. ...

If you’re still typing ssh -i ~/.ssh/my-key.pem -p 2222 admin@192.168.1.50 every time you connect, you’re doing it wrong. The SSH config file is one of the most underutilized productivity tools in a developer’s arsenal. The Basics: ~/.ssh/config Create or edit ~/.ssh/config: 1 2 3 4 5 Host dev HostName dev.example.com User deploy IdentityFile ~/.ssh/deploy_key Port 22 Now you just type ssh dev. That’s it. Host Patterns Wildcards let you apply settings to multiple hosts: ...

When GitHub-hosted runners aren’t enough—when you need GPU access, specific hardware, private network connectivity, or just want to stop paying per-minute—self-hosted runners are the answer. Why Self-Hosted? Performance: Your hardware, your speed. No cold starts, local caching, faster artifact access. Cost: After a certain threshold, self-hosted is dramatically cheaper. GitHub-hosted minutes add up fast for active repos. Access: Private networks, internal services, specialized hardware, air-gapped environments. Control: Exact OS versions, pre-installed dependencies, custom security configurations. ...

rsync synchronizes files between locations — local to local, local to remote, remote to local. It’s smart: it only transfers what’s changed, making it fast for incremental backups and deployments. Basic Syntax 1 rsync [options] source destination Local Sync 1 2 3 4 5 6 7 8 9 # Copy directory rsync -av /source/dir/ /dest/dir/ # Copy directory (trailing slash matters!) rsync -av /source/dir /dest/ # Creates /dest/dir/ rsync -av /source/dir/ /dest/ # Contents into /dest/ # Dry run (show what would happen) rsync -avn /source/ /dest/ Remote Sync (SSH) 1 2 3 4 5 6 7 8 9 10 11 # Local to remote rsync -av /local/dir/ user@remote:/remote/dir/ # Remote to local rsync -av user@remote:/remote/dir/ /local/dir/ # Different SSH port rsync -av -e 'ssh -p 2222' /local/ user@remote:/remote/ # With SSH key rsync -av -e 'ssh -i ~/.ssh/mykey' /local/ user@remote:/remote/ Common Options 1 2 3 4 5 6 7 8 9 -a, --archive Archive mode (preserves permissions, timestamps, etc.) -v, --verbose Verbose output -n, --dry-run Show what would be transferred -z, --compress Compress during transfer -P Progress + partial (resume interrupted transfers) --progress Show progress --delete Delete files in dest not in source -r, --recursive Recurse into directories -h, --human-readable Human-readable sizes Archive Mode (-a) -a is equivalent to -rlptgoD: ...

Cron has been scheduling tasks on Unix systems since 1975. It’s simple, reliable, and available everywhere. But that simplicity hides gotchas that break jobs in production. Cron Syntax ┌ │ │ │ │ │ ─ ─ ┌ │ │ │ │ ─ ─ ─ ─ ┌ │ │ │ ─ ─ ─ ─ ─ ─ ┌ │ │ ─ ─ ─ ─ ─ ─ ─ ─ ┌ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ c ─ ─ ─ ─ ─ o ─ ─ ─ ─ ─ m ─ ─ ─ ─ ─ m ─ ─ ─ ─ a m ─ ─ ─ ─ n i ─ ─ ─ d n h ─ ─ ─ u o ─ ─ t u d ─ ─ e r a ─ y m ─ ( ( o 0 0 o n d - - f t a 5 2 h y 9 3 m ) ) o ( o n 1 f t - h 1 w 2 e ( ) e 1 k - 3 ( 1 0 ) - 7 , 0 a n d 7 a r e S u n d a y ) Common Schedules 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Every minute * * * * * /path/to/script.sh # Every hour at minute 0 0 * * * * /path/to/script.sh # Every day at midnight 0 0 * * * /path/to/script.sh # Every day at 2:30 AM 30 2 * * * /path/to/script.sh # Every Monday at 9 AM 0 9 * * 1 /path/to/script.sh # Every 15 minutes */15 * * * * /path/to/script.sh # Every weekday at 6 PM 0 18 * * 1-5 /path/to/script.sh # First day of every month at midnight 0 0 1 * * /path/to/script.sh Special Strings 1 2 3 4 5 6 7 8 @reboot # Run once at startup @yearly # 0 0 1 1 * @annually # Same as @yearly @monthly # 0 0 1 * * @weekly # 0 0 * * 0 @daily # 0 0 * * * @midnight # Same as @daily @hourly # 0 * * * * Editing Crontabs 1 2 3 4 5 6 7 8 9 10 11 # Edit current user's crontab crontab -e # List current user's crontab crontab -l # Edit another user's crontab (as root) crontab -u username -e # Remove all cron jobs (careful!) crontab -r System Crontabs 1 2 3 4 5 6 7 8 9 10 11 # System-wide crontab (includes user field) /etc/crontab # Drop-in directory (no user field needed) /etc/cron.d/ # Periodic directories (scripts run by run-parts) /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /etc/crontab format includes username: ...

If you’re still typing ssh -i ~/.ssh/prod-key.pem -p 2222 ubuntu@ec2-54-123-45-67.compute-1.amazonaws.com, you’re working too hard. SSH config files let you define aliases with all your connection details. Basic Config Create or edit ~/.ssh/config: H o s t H U P I p o s o d r s e r e o t r t n d N t a u 2 i m b 2 t e u 2 y n 2 F e t i c u l 2 e - 5 ~ 4 / - . 1 s 2 s 3 h - / 4 p 5 r - o 6 d 7 - . k c e o y m . p p u e t m e - 1 . a m a z o n a w s . c o m Now connect with just: ...