Posts

Bash scripts have a reputation for being fragile, unreadable hacks. They don’t have to be. These patterns will help you write scripts that are maintainable, debuggable, and safe to run in production. The Essentials: Start Every Script Right 1 2 3 4 #!/usr/bin/env bash set -euo pipefail IFS=$'\n\t' What these do: set -e — Exit on any error set -u — Error on undefined variables set -o pipefail — Catch errors in pipelines IFS=$'\n\t' — Safer word splitting (no spaces) Script Template 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 #!/usr/bin/env bash set -euo pipefail # Script metadata readonly SCRIPT_NAME="$(basename "$0")" readonly SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" readonly SCRIPT_VERSION="1.0.0" # Default values VERBOSE=false DRY_RUN=false CONFIG_FILE="" # Colors (if terminal supports them) if [[ -t 1 ]]; then readonly RED='\033[0;31m' readonly GREEN='\033[0;32m' readonly YELLOW='\033[0;33m' readonly NC='\033[0m' # No Color else readonly RED='' GREEN='' YELLOW='' NC='' fi # Logging functions log_info() { echo -e "${GREEN}[INFO]${NC} $*"; } log_warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; } log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; } log_debug() { [[ "$VERBOSE" == true ]] && echo -e "[DEBUG] $*" || true; } # Cleanup function cleanup() { local exit_code=$? # Add cleanup tasks here log_debug "Cleaning up..." exit $exit_code } trap cleanup EXIT # Usage usage() { cat <<EOF Usage: $SCRIPT_NAME [OPTIONS] <argument> Description of what this script does. Options: -h, --help Show this help message -v, --verbose Enable verbose output -n, --dry-run Show what would be done -c, --config FILE Path to config file --version Show version Examples: $SCRIPT_NAME -v input.txt $SCRIPT_NAME --config /etc/myapp.conf data/ EOF } # Parse arguments parse_args() { while [[ $# -gt 0 ]]; do case $1 in -h|--help) usage exit 0 ;; -v|--verbose) VERBOSE=true shift ;; -n|--dry-run) DRY_RUN=true shift ;; -c|--config) CONFIG_FILE="$2" shift 2 ;; --version) echo "$SCRIPT_NAME version $SCRIPT_VERSION" exit 0 ;; -*) log_error "Unknown option: $1" usage exit 1 ;; *) ARGS+=("$1") shift ;; esac done } # Validation validate() { if [[ ${#ARGS[@]} -eq 0 ]]; then log_error "Missing required argument" usage exit 1 fi if [[ -n "$CONFIG_FILE" && ! -f "$CONFIG_FILE" ]]; then log_error "Config file not found: $CONFIG_FILE" exit 1 fi } # Main logic main() { log_info "Starting $SCRIPT_NAME" log_debug "Arguments: ${ARGS[*]}" for arg in "${ARGS[@]}"; do if [[ "$DRY_RUN" == true ]]; then log_info "[DRY RUN] Would process: $arg" else log_info "Processing: $arg" # Actual work here fi done log_info "Done" } # Entry point ARGS=() parse_args "$@" validate main Error Handling Check Command Existence 1 2 3 4 5 6 7 8 9 10 require_command() { if ! command -v "$1" &> /dev/null; then log_error "Required command not found: $1" exit 1 fi } require_command curl require_command jq require_command docker Retry Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 retry() { local max_attempts=$1 local delay=$2 shift 2 local cmd=("$@") local attempt=1 while (( attempt <= max_attempts )); do if "${cmd[@]}"; then return 0 fi log_warn "Attempt $attempt/$max_attempts failed, retrying in ${delay}s..." sleep "$delay" ((attempt++)) done log_error "Command failed after $max_attempts attempts: ${cmd[*]}" return 1 } # Usage retry 3 5 curl -sf https://api.example.com/health Timeout Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 with_timeout() { local timeout=$1 shift timeout "$timeout" "$@" || { local exit_code=$? if [[ $exit_code -eq 124 ]]; then log_error "Command timed out after ${timeout}s" fi return $exit_code } } # Usage with_timeout 30 curl -s https://api.example.com Safe File Operations Temporary Files 1 2 3 4 5 6 7 # Create temp file that's automatically cleaned up TEMP_FILE=$(mktemp) trap 'rm -f "$TEMP_FILE"' EXIT # Or temp directory TEMP_DIR=$(mktemp -d) trap 'rm -rf "$TEMP_DIR"' EXIT Safe Write (Atomic) 1 2 3 4 5 6 7 8 9 10 11 12 safe_write() { local target=$1 local content=$2 local temp_file temp_file=$(mktemp) echo "$content" > "$temp_file" mv "$temp_file" "$target" } # Usage safe_write /etc/myapp/config.json '{"key": "value"}' Backup Before Modify 1 2 3 4 5 6 7 8 9 backup_file() { local file=$1 local backup="${file}.bak.$(date +%Y%m%d_%H%M%S)" if [[ -f "$file" ]]; then cp "$file" "$backup" log_info "Backup created: $backup" fi } Input Validation File Checks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 validate_file() { local file=$1 [[ -e "$file" ]] || { log_error "File not found: $file"; return 1; } [[ -f "$file" ]] || { log_error "Not a regular file: $file"; return 1; } [[ -r "$file" ]] || { log_error "File not readable: $file"; return 1; } } validate_directory() { local dir=$1 [[ -e "$dir" ]] || { log_error "Directory not found: $dir"; return 1; } [[ -d "$dir" ]] || { log_error "Not a directory: $dir"; return 1; } [[ -w "$dir" ]] || { log_error "Directory not writable: $dir"; return 1; } } Input Sanitization 1 2 3 4 5 6 7 8 sanitize_filename() { local input=$1 # Remove path components and dangerous characters echo "${input##*/}" | tr -cd '[:alnum:]._-' } # Usage safe_name=$(sanitize_filename "$user_input") Configuration Config File Loading 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 load_config() { local config_file=$1 if [[ -f "$config_file" ]]; then log_debug "Loading config: $config_file" # shellcheck source=/dev/null source "$config_file" else log_warn "Config file not found, using defaults: $config_file" fi } # Config file format (config.sh): # DB_HOST="localhost" # DB_PORT=5432 # ENABLE_FEATURE=true Environment Variable Defaults 1 2 3 4 5 6 : "${DB_HOST:=localhost}" : "${DB_PORT:=5432}" : "${LOG_LEVEL:=info}" # Or with validation DB_HOST="${DB_HOST:?DB_HOST environment variable required}" Parallel Execution Using xargs 1 2 3 4 process_files() { find . -name "*.txt" -print0 | \ xargs -0 -P 4 -I {} bash -c 'process_single_file "$1"' _ {} } Using Background Jobs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 parallel_tasks() { local pids=() for item in "${items[@]}"; do process_item "$item" & pids+=($!) done # Wait for all and check results local failed=0 for pid in "${pids[@]}"; do if ! wait "$pid"; then ((failed++)) fi done [[ $failed -eq 0 ]] || log_error "$failed tasks failed" return $failed } Locking Prevent Concurrent Runs 1 2 3 4 5 6 7 8 9 10 11 LOCK_FILE="/var/run/${SCRIPT_NAME}.lock" acquire_lock() { if ! mkdir "$LOCK_FILE" 2>/dev/null; then log_error "Another instance is running (lock: $LOCK_FILE)" exit 1 fi trap 'rm -rf "$LOCK_FILE"' EXIT } acquire_lock With flock 1 2 3 4 5 6 7 8 9 10 11 12 LOCK_FD=200 LOCK_FILE="/var/run/${SCRIPT_NAME}.lock" acquire_lock() { eval "exec $LOCK_FD>$LOCK_FILE" if ! flock -n $LOCK_FD; then log_error "Another instance is running" exit 1 fi } acquire_lock Testing Assertions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 assert_equals() { local expected=$1 actual=$2 message=${3:-"Assertion failed"} if [[ "$expected" != "$actual" ]]; then log_error "$message: expected '$expected', got '$actual'" return 1 fi } assert_file_exists() { local file=$1 if [[ ! -f "$file" ]]; then log_error "File should exist: $file" return 1 fi } # Usage assert_equals "200" "$status_code" "HTTP status check" assert_file_exists "/tmp/output.txt" Test Mode 1 2 3 4 5 6 7 if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then # Script is being run directly main "$@" else # Script is being sourced (for testing) log_debug "Script sourced, not executing main" fi Debugging Debug Mode 1 [[ "${DEBUG:-}" == true ]] && set -x Trace Function Calls 1 2 3 4 5 6 7 8 trace() { log_debug "TRACE: ${FUNCNAME[1]}(${*})" } my_function() { trace "$@" # function body } Common Pitfalls Quote Everything 1 2 3 4 5 # Bad if [ $var == "value" ]; then # Good if [[ "$var" == "value" ]]; then Use Arrays for Lists 1 2 3 4 5 6 7 # Bad files="file1.txt file2.txt file with spaces.txt" for f in $files; do echo "$f"; done # Good files=("file1.txt" "file2.txt" "file with spaces.txt") for f in "${files[@]}"; do echo "$f"; done Check Command Success 1 2 3 4 5 6 7 8 9 10 11 # Bad output=$(some_command) echo "$output" # Good if output=$(some_command 2>&1); then echo "$output" else log_error "Command failed: $output" exit 1 fi Good bash scripts are defensive, verbose when needed, and fail fast. Start with the template, add what you need, and resist the urge to be clever. ...

Docker Compose bridges the gap between single-container development and full orchestration. These patterns will help you build maintainable, production-ready configurations. Project Structure m ├ ├ ├ ├ ├ ├ └ y ─ ─ ─ ─ ─ ─ ─ p ─ ─ ─ ─ ─ ─ ─ r o d d d d . . s ├ │ ├ │ └ j o o o o e e e ─ ─ ─ e c c c c n n r ─ ─ ─ c k k k k v v v t e e e e . i a └ w └ n ├ └ / r r r r e c p ─ o ─ g ─ ─ - - - - x e p ─ r ─ i ─ ─ c c c c a s k n o o o o m D e D x D n m m m m p o r / o g p p p p l c c c i o o o o e k k k n s s s s e e e x e e e e r r r . . . . . f f f c y o p t i i i o m v r e l l l n l e o s e e e f r d t r . . i y y d m m e l l . y m l # # # # # # B D P T E T a e r e n e s v o s v m e d t i p u r l c v c c o a o e t o n t n r i n m e f r o f e i i n i n ( g d g t c u e u o r s v r v m a e a a m t ( r t r i i a r i i t o u i o a t n t d n b e o e l d - s e ) l s o a d e d ) Base Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 # docker-compose.yml version: "3.8" services: app: build: context: ./services/app dockerfile: Dockerfile environment: - DATABASE_URL=${DATABASE_URL} - REDIS_URL=${REDIS_URL} depends_on: db: condition: service_healthy redis: condition: service_started networks: - backend restart: unless-stopped db: image: postgres:15-alpine environment: POSTGRES_DB: ${DB_NAME} POSTGRES_USER: ${DB_USER} POSTGRES_PASSWORD: ${DB_PASSWORD} volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"] interval: 10s timeout: 5s retries: 5 networks: - backend redis: image: redis:7-alpine volumes: - redis_data:/data networks: - backend networks: backend: driver: bridge volumes: postgres_data: redis_data: Development Overrides 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 # docker-compose.override.yml (auto-loaded with docker-compose up) version: "3.8" services: app: build: target: development volumes: - ./src:/app/src:cached - /app/node_modules ports: - "3000:3000" - "9229:9229" # Debugger environment: - DEBUG=true - LOG_LEVEL=debug command: npm run dev db: ports: - "5432:5432" redis: ports: - "6379:6379" Production Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 # docker-compose.prod.yml version: "3.8" services: app: build: target: production deploy: replicas: 3 resources: limits: cpus: '1' memory: 512M reservations: cpus: '0.5' memory: 256M restart_policy: condition: on-failure delay: 5s max_attempts: 3 environment: - NODE_ENV=production - LOG_LEVEL=info logging: driver: json-file options: max-size: "10m" max-file: "3" nginx: image: nginx:alpine ports: - "80:80" - "443:443" volumes: - ./services/nginx/nginx.conf:/etc/nginx/nginx.conf:ro - ./certs:/etc/nginx/certs:ro depends_on: - app networks: - backend - frontend networks: frontend: driver: bridge Run with: ...

You know git add, commit, push, and pull. That gets you through 90% of daily work. But the remaining 10%—untangling merge conflicts, finding bug introductions, managing multiple branches simultaneously—requires deeper knowledge. Interactive Rebase The most powerful tool for cleaning up history before sharing. 1 2 3 4 5 # Rebase last 5 commits git rebase -i HEAD~5 # Rebase onto main git rebase -i main The editor opens with commits listed oldest-first: ...

curl is the universal language of HTTP. Every API doc includes curl examples. Every debugging session starts with “can you curl it?” If you’re not comfortable with curl, you’re missing the most portable tool in your kit. Basic Requests 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # GET (default) curl https://api.example.com/users # With headers shown curl -i https://api.example.com/users # Headers only curl -I https://api.example.com/users # Silent (no progress bar) curl -s https://api.example.com/users # Follow redirects curl -L https://example.com/redirect # Verbose (debug mode) curl -v https://api.example.com/users HTTP Methods 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # POST curl -X POST https://api.example.com/users # PUT curl -X PUT https://api.example.com/users/1 # PATCH curl -X PATCH https://api.example.com/users/1 # DELETE curl -X DELETE https://api.example.com/users/1 # HEAD (headers only, like -I) curl -X HEAD https://api.example.com/users Sending Data Form Data 1 2 3 4 5 6 7 # URL-encoded form curl -X POST https://api.example.com/login \ -d "username=admin&password=secret" # From file curl -X POST https://api.example.com/login \ -d @credentials.txt JSON Data 1 2 3 4 5 6 7 8 9 10 11 12 # Inline JSON curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d '{"name":"Alice","email":"alice@example.com"}' # From file curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d @user.json # Using --json (curl 7.82+) curl --json '{"name":"Alice"}' https://api.example.com/users File Upload 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Single file curl -X POST https://api.example.com/upload \ -F "file=@document.pdf" # Multiple files curl -X POST https://api.example.com/upload \ -F "file1=@doc1.pdf" \ -F "file2=@doc2.pdf" # File with custom filename curl -X POST https://api.example.com/upload \ -F "file=@localname.pdf;filename=remote.pdf" # File with content type curl -X POST https://api.example.com/upload \ -F "file=@image.png;type=image/png" # Mixed form data and files curl -X POST https://api.example.com/upload \ -F "title=My Document" \ -F "file=@document.pdf" Headers 1 2 3 4 5 6 7 8 9 10 11 12 13 # Custom header curl -H "X-Custom-Header: value" https://api.example.com # Multiple headers curl -H "Accept: application/json" \ -H "X-API-Version: 2" \ https://api.example.com # User agent curl -A "MyApp/1.0" https://api.example.com # Referer curl -e "https://example.com" https://api.example.com Authentication Basic Auth 1 2 3 4 5 6 7 8 # Username and password curl -u username:password https://api.example.com # Prompt for password curl -u username https://api.example.com # In URL (not recommended) curl https://username:password@api.example.com Bearer Token 1 curl -H "Authorization: Bearer YOUR_TOKEN" https://api.example.com API Key 1 2 3 4 5 # In header curl -H "X-API-Key: YOUR_KEY" https://api.example.com # In query string curl "https://api.example.com?api_key=YOUR_KEY" OAuth 2.0 Flow 1 2 3 4 5 6 7 8 9 # Get access token curl -X POST https://auth.example.com/oauth/token \ -d "grant_type=client_credentials" \ -d "client_id=YOUR_CLIENT_ID" \ -d "client_secret=YOUR_SECRET" # Use token TOKEN="eyJ..." curl -H "Authorization: Bearer $TOKEN" https://api.example.com/resource Digest Auth 1 curl --digest -u username:password https://api.example.com Output Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Save to file curl -o output.html https://example.com # Save with remote filename curl -O https://example.com/file.zip # Save multiple files curl -O https://example.com/file1.zip -O https://example.com/file2.zip # Append to file curl https://example.com >> output.txt # Write headers to file curl -D headers.txt https://example.com # Output to stdout and file curl https://example.com | tee output.html Timeouts and Retries 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Connection timeout (seconds) curl --connect-timeout 5 https://api.example.com # Max time for entire operation curl -m 30 https://api.example.com # Retry on failure curl --retry 3 https://api.example.com # Retry with delay curl --retry 3 --retry-delay 5 https://api.example.com # Retry on specific errors curl --retry 3 --retry-all-errors https://api.example.com SSL/TLS 1 2 3 4 5 6 7 8 9 10 11 12 # Skip certificate verification (insecure!) curl -k https://self-signed.example.com # Use specific CA certificate curl --cacert /path/to/ca.crt https://api.example.com # Client certificate curl --cert client.crt --key client.key https://api.example.com # Force TLS version curl --tlsv1.2 https://api.example.com curl --tlsv1.3 https://api.example.com Proxy 1 2 3 4 5 6 7 8 9 10 11 # HTTP proxy curl -x http://proxy:8080 https://api.example.com # SOCKS5 proxy curl --socks5 localhost:1080 https://api.example.com # Proxy with auth curl -x http://user:pass@proxy:8080 https://api.example.com # No proxy for specific hosts curl --noproxy "localhost,*.internal" https://api.example.com Cookies 1 2 3 4 5 6 7 8 9 10 11 # Send cookie curl -b "session=abc123" https://api.example.com # Send cookies from file curl -b cookies.txt https://api.example.com # Save cookies to file curl -c cookies.txt https://api.example.com/login # Full session (save and send) curl -b cookies.txt -c cookies.txt https://api.example.com Response Inspection 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # HTTP status code only curl -s -o /dev/null -w "%{http_code}" https://api.example.com # Response time curl -s -o /dev/null -w "%{time_total}s" https://api.example.com # Detailed timing curl -s -o /dev/null -w " DNS: %{time_namelookup}s Connect: %{time_connect}s TLS: %{time_appconnect}s Start: %{time_starttransfer}s Total: %{time_total}s Size: %{size_download} bytes Speed: %{speed_download} bytes/sec " https://api.example.com # Content type curl -s -o /dev/null -w "%{content_type}" https://api.example.com Scripting Patterns Health Check 1 2 3 4 5 6 7 8 9 10 11 #!/bin/bash URL="https://api.example.com/health" STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$URL") if [ "$STATUS" -eq 200 ]; then echo "OK" exit 0 else echo "FAIL: HTTP $STATUS" exit 1 fi API Wrapper 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #!/bin/bash API_BASE="https://api.example.com" API_KEY="${API_KEY:?API_KEY required}" api_get() { curl -s -H "Authorization: Bearer $API_KEY" "$API_BASE$1" } api_post() { curl -s -X POST \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$2" \ "$API_BASE$1" } # Usage api_get "/users" | jq '.' api_post "/users" '{"name":"Alice"}' Retry with Backoff 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 #!/bin/bash URL="$1" MAX_RETRIES=5 RETRY_DELAY=1 for i in $(seq 1 $MAX_RETRIES); do RESPONSE=$(curl -s -w "\n%{http_code}" "$URL") STATUS=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') if [ "$STATUS" -eq 200 ]; then echo "$BODY" exit 0 fi echo "Attempt $i failed (HTTP $STATUS), retrying in ${RETRY_DELAY}s..." >&2 sleep $RETRY_DELAY RETRY_DELAY=$((RETRY_DELAY * 2)) done echo "Failed after $MAX_RETRIES attempts" >&2 exit 1 Parallel Requests 1 2 3 4 5 # Using xargs cat urls.txt | xargs -P 10 -I {} curl -s -o /dev/null -w "{}: %{http_code}\n" {} # Using GNU parallel parallel -j 10 curl -s -o /dev/null -w "{}: %{http_code}\n" ::: $(cat urls.txt) Config Files Create ~/.curlrc for defaults: ...

Every API returns JSON. Every config file is JSON. If you’re not fluent in jq, you’re copying data by hand like it’s 1995. The Basics 1 2 3 4 5 6 7 8 9 10 # Pretty print echo '{"name":"test","value":42}' | jq '.' # Extract a field echo '{"name":"test","value":42}' | jq '.name' # "test" # Raw output (no quotes) echo '{"name":"test","value":42}' | jq -r '.name' # test Working with APIs 1 2 3 4 5 6 7 8 # GitHub API curl -s https://api.github.com/users/torvalds | jq '.login, .public_repos' # Extract specific fields curl -s https://api.github.com/repos/stedolan/jq | jq '{name, stars: .stargazers_count, language}' # AWS CLI (already outputs JSON) aws ec2 describe-instances | jq '.Reservations[].Instances[] | {id: .InstanceId, state: .State.Name}' Array Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # Sample data DATA='[{"name":"alice","age":30},{"name":"bob","age":25},{"name":"carol","age":35}]' # First element echo $DATA | jq '.[0]' # Last element echo $DATA | jq '.[-1]' # Slice echo $DATA | jq '.[0:2]' # All names echo $DATA | jq '.[].name' # Array of names echo $DATA | jq '[.[].name]' # Length echo $DATA | jq 'length' Filtering 1 2 3 4 5 6 7 8 9 10 11 # Select by condition echo $DATA | jq '.[] | select(.age > 28)' # Multiple conditions echo $DATA | jq '.[] | select(.age > 25 and .name != "carol")' # Contains echo '[{"tags":["web","api"]},{"tags":["cli"]}]' | jq '.[] | select(.tags | contains(["api"]))' # Has key echo '{"a":1,"b":null}' | jq 'has("a"), has("c")' Transformation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Add/modify fields echo '{"name":"test"}' | jq '. + {status: "active", count: 0}' # Update existing field echo '{"count":5}' | jq '.count += 1' # Delete field echo '{"a":1,"b":2,"c":3}' | jq 'del(.b)' # Rename key echo '{"old_name":"value"}' | jq '{new_name: .old_name}' # Map over array echo '[1,2,3,4,5]' | jq 'map(. * 2)' # Map with objects echo $DATA | jq 'map({username: .name, birth_year: (2026 - .age)})' String Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Concatenation echo '{"first":"John","last":"Doe"}' | jq '.first + " " + .last' # String interpolation echo '{"name":"test","ver":"1.0"}' | jq '"\(.name)-\(.ver).tar.gz"' # Split echo '{"path":"/usr/local/bin"}' | jq '.path | split("/")' # Join echo '["a","b","c"]' | jq 'join(",")' # Upper/lower echo '"Hello World"' | jq 'ascii_downcase' echo '"Hello World"' | jq 'ascii_upcase' # Test regex echo '{"email":"test@example.com"}' | jq '.email | test("@")' # Replace echo '"hello world"' | jq 'gsub("world"; "jq")' Conditionals 1 2 3 4 5 6 7 8 9 10 11 # If-then-else echo '{"status":200}' | jq 'if .status == 200 then "ok" else "error" end' # Alternative operator (default value) echo '{"a":1}' | jq '.b // "default"' # Null handling echo '{"a":null}' | jq '.a // "was null"' # Error handling echo '{}' | jq '.missing.nested // "not found"' Grouping and Aggregation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 LOGS='[ {"level":"error","msg":"failed"}, {"level":"info","msg":"started"}, {"level":"error","msg":"timeout"}, {"level":"info","msg":"completed"} ]' # Group by field echo $LOGS | jq 'group_by(.level)' # Count per group echo $LOGS | jq 'group_by(.level) | map({level: .[0].level, count: length})' # Unique values echo $LOGS | jq '[.[].level] | unique' # Sort echo $DATA | jq 'sort_by(.age)' # Reverse sort echo $DATA | jq 'sort_by(.age) | reverse' # Min/max echo '[5,2,8,1,9]' | jq 'min, max' # Sum echo '[1,2,3,4,5]' | jq 'add' # Average echo '[1,2,3,4,5]' | jq 'add / length' Constructing Output 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Build new object curl -s https://api.github.com/users/torvalds | jq '{ username: .login, repos: .public_repos, profile: .html_url }' # Build array echo '{"users":[{"name":"a"},{"name":"b"}]}' | jq '[.users[].name]' # Multiple outputs to array echo '{"a":1,"b":2}' | jq '[.a, .b, .a + .b]' # Key-value pairs echo '{"a":1,"b":2}' | jq 'to_entries' # [{"key":"a","value":1},{"key":"b","value":2}] # Back to object echo '[{"key":"a","value":1}]' | jq 'from_entries' # Transform keys echo '{"old_a":1,"old_b":2}' | jq 'with_entries(.key |= ltrimstr("old_"))' Real-World Examples Parse AWS Instance List 1 2 3 4 5 aws ec2 describe-instances | jq -r ' .Reservations[].Instances[] | [.InstanceId, .State.Name, (.Tags[]? | select(.Key=="Name") | .Value) // "unnamed"] | @tsv ' Filter Docker Containers 1 2 3 4 5 6 docker inspect $(docker ps -q) | jq '.[] | { name: .Name, image: .Config.Image, status: .State.Status, ip: .NetworkSettings.IPAddress }' Process Log Files 1 2 3 4 5 6 7 # Count errors by type cat app.log | jq -s 'group_by(.error_type) | map({type: .[0].error_type, count: length}) | sort_by(.count) | reverse' # Extract errors from last hour cat app.log | jq --arg cutoff "$(date -d '1 hour ago' -Iseconds)" ' select(.timestamp > $cutoff and .level == "error") ' Transform Config Files 1 2 3 4 5 6 7 8 # Merge configs jq -s '.[0] * .[1]' base.json override.json # Update nested value jq '.database.host = "newhost.example.com"' config.json # Add to array jq '.allowed_ips += ["10.0.0.5"]' config.json Generate Reports 1 2 3 4 5 6 # Kubernetes pod status kubectl get pods -o json | jq -r ' .items[] | [.metadata.name, .status.phase, (.status.containerStatuses[0].restartCount // 0)] | @tsv ' | column -t Useful Flags 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 # Compact output (no pretty print) jq -c '.' # Raw output (no quotes on strings) jq -r '.name' # Raw input (treat input as string, not JSON) jq -R 'split(",")' # Slurp (read all inputs into array) cat *.json | jq -s '.' # Pass variable jq --arg name "test" '.name = $name' # Pass JSON variable jq --argjson count 42 '.count = $count' # Read from file jq --slurpfile users users.json '.users = $users' # Exit with error if output is null/false jq -e '.important_field' && echo "exists" # Sort keys in output jq -S '.' Output Formats 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Tab-separated echo $DATA | jq -r '.[] | [.name, .age] | @tsv' # CSV echo $DATA | jq -r '.[] | [.name, .age] | @csv' # URI encoding echo '{"q":"hello world"}' | jq -r '.q | @uri' # Base64 echo '{"data":"secret"}' | jq -r '.data | @base64' # Shell-safe echo '{"cmd":"echo hello"}' | jq -r '.cmd | @sh' Debugging 1 2 3 4 5 6 7 8 9 10 11 # Show type echo '{"a":[1,2,3]}' | jq '.a | type' # Show keys echo '{"a":1,"b":2}' | jq 'keys' # Debug output (shows intermediate values) echo '{"x":{"y":{"z":1}}}' | jq '.x | debug | .y | debug | .z' # Path to value echo '{"a":{"b":{"c":1}}}' | jq 'path(.. | select(. == 1))' Quick Reference 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 # Identity . # Field access .field .field.nested # Array access .[0] .[-1] .[2:5] # Iterate array .[] # Pipe .[] | .name # Collect into array [.[] | .name] # Object construction {newkey: .oldkey} # Conditionals if COND then A else B end VALUE // DEFAULT # Comparison ==, !=, <, >, <=, >= and, or, not # Array functions map(f), select(f), sort_by(f), group_by(f), unique, length, first, last, nth(n), flatten, reverse, contains(x), inside(x), add, min, max # String functions split(s), join(s), test(re), match(re), gsub(re;s), ascii_downcase, ascii_upcase, ltrimstr(s), rtrimstr(s), startswith(s), endswith(s) # Object functions keys, values, has(k), in(o), to_entries, from_entries, with_entries(f) # Type functions type, isnumber, isstring, isnull, isboolean, isarray, isobject jq turns JSON from a data format into a query language. Once you internalize the pipe-and-filter model, you’ll wonder how you ever survived without it. ...

Cron has run scheduled tasks since 1975. It works, but systemd timers offer significant advantages: integrated logging, dependency management, randomized delays, and calendar-based scheduling that actually makes sense. Why Switch from Cron? Logging: Timer output goes to journald. No more digging through mail or custom log files. Dependencies: Wait for network, mounts, or other services before running. Accuracy: Monotonic timers don’t drift. Calendar timers handle DST correctly. Visibility: systemctl list-timers shows all scheduled jobs and when they’ll run next. ...

The third factor of the 12-Factor App methodology states: “Store config in the environment.” Simple advice that’s surprisingly easy to get wrong. The Core Principle Configuration that varies between environments (dev, staging, production) should come from environment variables, not code. This includes: Database connection strings API keys and secrets Feature flags Service URLs Port numbers Log levels What stays in code: application logic, default behaviors, anything that doesn’t change between deploys. ...

Make was designed for compiling C programs in 1976. Nearly 50 years later, it’s still one of the most practical automation tools available—not for its original purpose, but as a universal task runner. Why Make in 2026? It’s already installed. Every Unix system has make. No npm install, no pip, no version managers. It’s declarative. Define what you want, not how to get there (with dependencies handled automatically). It’s documented. make help can list all your targets. The Makefile itself is documentation. ...

If you’re still typing ssh -i ~/.ssh/my-key.pem -p 2222 admin@192.168.1.50 every time you connect, you’re doing it wrong. The SSH config file is one of the most underutilized productivity tools in a developer’s arsenal. The Basics: ~/.ssh/config Create or edit ~/.ssh/config: 1 2 3 4 5 Host dev HostName dev.example.com User deploy IdentityFile ~/.ssh/deploy_key Port 22 Now you just type ssh dev. That’s it. Host Patterns Wildcards let you apply settings to multiple hosts: ...

When GitHub-hosted runners aren’t enough—when you need GPU access, specific hardware, private network connectivity, or just want to stop paying per-minute—self-hosted runners are the answer. Why Self-Hosted? Performance: Your hardware, your speed. No cold starts, local caching, faster artifact access. Cost: After a certain threshold, self-hosted is dramatically cheaper. GitHub-hosted minutes add up fast for active repos. Access: Private networks, internal services, specialized hardware, air-gapped environments. Control: Exact OS versions, pre-installed dependencies, custom security configurations. ...