Devops

PostgreSQL is fast out of the box. But “fast enough for development” and “fast enough for production” are different conversations. These techniques will help you find and fix performance bottlenecks. Finding Slow Queries Enable Query Logging 1 2 3 4 5 6 -- Log queries slower than 500ms ALTER SYSTEM SET log_min_duration_statement = '500ms'; SELECT pg_reload_conf(); -- Check current setting SHOW log_min_duration_statement; pg_stat_statements Extension The most valuable performance tool: ...

Ansible’s simplicity is deceptive. Anyone can write a playbook that works once. Writing playbooks that work reliably, repeatedly, and maintainably requires discipline and patterns. Project Structure a ├ ├ │ │ │ │ │ │ │ │ ├ │ │ │ ├ │ │ │ └ n ─ ─ ─ ─ ─ s ─ ─ ─ ─ ─ i b a i ├ │ │ │ │ └ p ├ ├ └ r ├ ├ └ f └ l n n ─ ─ l ─ ─ ─ o ─ ─ ─ i ─ e s v ─ ─ a ─ ─ ─ l ─ ─ ─ l ─ / i e y e e b n p ├ └ s ├ └ b s w d s c n p s s l t r ─ ─ t ─ ─ o i e a / o g o / c e o o ─ ─ a ─ ─ o t b t m i s r . r d g k e s a m n t i c y u h g ├ └ i h g s . e b o x g p f / c o r ─ ─ n o r y r a n r t g t s o ─ ─ g s m v s e s i t u / t u l e e s / o s p a w s p r s q n . _ l e . _ s . l y v l b y v . y / m a . s m a y m l r y e l r m l s m r s l / l v / e r s . y m l Inventory Best Practices 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # inventory/production/hosts.yml all: children: webservers: hosts: web1.example.com: web2.example.com: vars: http_port: 80 databases: hosts: db1.example.com: postgres_role: primary db2.example.com: postgres_role: replica loadbalancers: hosts: lb1.example.com: vars: ansible_user: deploy ansible_python_interpreter: /usr/bin/python3 Role Structure r ├ │ ├ │ ├ │ │ │ ├ │ ├ │ ├ │ ├ │ └ o ─ ─ ─ ─ ─ ─ ─ ─ l ─ ─ ─ ─ ─ ─ ─ ─ e s d └ v └ t ├ ├ └ h └ t └ f └ m └ m └ / e ─ a ─ a ─ ─ ─ a ─ e ─ i ─ e ─ o ─ n f ─ r ─ s ─ ─ ─ n ─ m ─ l ─ t ─ l ─ g a s k d p e a e i u m / m s m i c l m l n s s / m c d n l a a / a n o e a a g / s a u e x t i i i s n r i t i l i l f / s n n n t f s n e n / n e a / . . . a i / . s x . / u y y y l g y / . y l m m m l u m c m t l l l . r l o l / y e n m . f l y . m j l 2 # # # # # # D R E S D T e o n e e e f l t r p s a e r v e t u y i n i l v c d n t a p e e g r o n v i i r c a a n e i r b t s e i l t s a e a b s r l t e ( s h h i a ( g n l h d o e l w r e e r s p s t r i p o r r i i o t r y i ) t y ) Task Patterns Always Name Tasks 1 2 3 4 5 6 7 8 9 10 # Bad - apt: name: nginx state: present # Good - name: Install nginx apt: name: nginx state: present Use Block for Related Tasks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 - name: Configure SSL block: - name: Copy SSL certificate copy: src: "{{ ssl_cert }}" dest: /etc/ssl/certs/ mode: '0644' - name: Copy SSL private key copy: src: "{{ ssl_key }}" dest: /etc/ssl/private/ mode: '0600' - name: Enable SSL site file: src: /etc/nginx/sites-available/ssl.conf dest: /etc/nginx/sites-enabled/ssl.conf state: link notify: Reload nginx when: ssl_enabled | bool Error Handling 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 - name: Deploy application block: - name: Pull latest code git: repo: "{{ app_repo }}" dest: "{{ app_path }}" version: "{{ app_version }}" register: git_result - name: Run migrations command: ./manage.py migrate args: chdir: "{{ app_path }}" when: git_result.changed rescue: - name: Rollback to previous version git: repo: "{{ app_repo }}" dest: "{{ app_path }}" version: "{{ previous_version }}" - name: Notify failure slack: token: "{{ slack_token }}" msg: "Deploy failed on {{ inventory_hostname }}" always: - name: Clean up temp files file: path: /tmp/deploy state: absent Variables Variable Precedence (use intentionally) 1 2 3 4 5 6 7 8 9 10 11 12 13 # defaults/main.yml - Easily overridden defaults nginx_worker_processes: auto nginx_worker_connections: 1024 # vars/main.yml - Role-specific constants nginx_user: www-data nginx_conf_path: /etc/nginx/nginx.conf # group_vars/webservers.yml - Group-specific nginx_worker_connections: 4096 # host_vars/web1.yml - Host-specific nginx_worker_processes: 4 Variable Validation 1 2 3 4 5 6 7 8 - name: Validate required variables assert: that: - app_version is defined - app_version | length > 0 - db_password is defined fail_msg: "Required variables are not set" success_msg: "All required variables present" Default Values 1 2 3 4 5 6 7 8 - name: Set configuration template: src: config.j2 dest: /etc/app/config.yml vars: max_connections: "{{ app_max_connections | default(100) }}" timeout: "{{ app_timeout | default(30) }}" debug: "{{ app_debug | default(false) | bool }}" Handlers 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # handlers/main.yml - name: Reload nginx service: name: nginx state: reloaded listen: "reload web server" - name: Restart nginx service: name: nginx state: restarted listen: "restart web server" # In tasks - name: Update nginx config template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf notify: "reload web server" Flush Handlers When Needed 1 2 3 4 5 6 7 8 9 10 11 12 13 14 - name: Install nginx apt: name: nginx state: present notify: Start nginx - name: Flush handlers meta: flush_handlers - name: Configure nginx template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf # nginx is now guaranteed to be running Conditionals Clean Conditionals 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Use bool filter for string booleans - name: Enable debug mode template: src: debug.conf.j2 dest: /etc/app/debug.conf when: debug_mode | bool # Check if variable is defined and not empty - name: Set custom config copy: content: "{{ custom_config }}" dest: /etc/app/custom.conf when: - custom_config is defined - custom_config | length > 0 # Multiple conditions - name: Deploy to production include_tasks: deploy.yml when: - env == 'production' - deploy_enabled | bool - inventory_hostname in groups['webservers'] Loops Modern Loop Syntax 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 # Simple loop - name: Install packages apt: name: "{{ item }}" state: present loop: - nginx - python3 - htop # Better: Install all at once - name: Install packages apt: name: - nginx - python3 - htop state: present # Loop with index - name: Create users user: name: "{{ item.name }}" uid: "{{ 1000 + index }}" groups: "{{ item.groups }}" loop: "{{ users }}" loop_control: index_var: index label: "{{ item.name }}" # Cleaner output # Dict loop - name: Configure services template: src: "{{ item.key }}.conf.j2" dest: "/etc/{{ item.key }}/config.conf" loop: "{{ services | dict2items }}" when: item.value.enabled | bool Templates Jinja2 Best Practices 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 {# templates/nginx.conf.j2 #} # Managed by Ansible - DO NOT EDIT # Last updated: {{ ansible_date_time.iso8601 }} # Host: {{ inventory_hostname }} worker_processes {{ nginx_worker_processes }}; events { worker_connections {{ nginx_worker_connections }}; } http { {% for server in nginx_servers %} server { listen {{ server.port | default(80) }}; server_name {{ server.name }}; {% if server.ssl | default(false) %} ssl_certificate {{ server.ssl_cert }}; ssl_certificate_key {{ server.ssl_key }}; {% endif %} {% for location in server.locations | default([]) %} location {{ location.path }} { {{ location.directive }}; } {% endfor %} } {% endfor %} } Template Validation 1 2 3 4 5 6 - name: Generate nginx config template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf validate: nginx -t -c %s notify: Reload nginx Idempotency Patterns Check Mode Support 1 2 3 4 5 6 7 8 9 10 11 - name: Get current version command: cat /opt/app/VERSION register: current_version changed_when: false check_mode: false # Always run, even in check mode - name: Deploy new version unarchive: src: "app-{{ target_version }}.tar.gz" dest: /opt/app/ when: current_version.stdout != target_version Custom Changed Conditions 1 2 3 4 5 - name: Run database migration command: ./manage.py migrate --check register: migration_check changed_when: "'No migrations to apply' not in migration_check.stdout" failed_when: migration_check.rc not in [0, 1] Avoid Command When Possible 1 2 3 4 5 6 7 8 9 10 # Bad - not idempotent - name: Create directory command: mkdir -p /opt/app # Good - idempotent - name: Create directory file: path: /opt/app state: directory mode: '0755' Secrets Management Ansible Vault 1 2 3 4 5 6 7 8 9 10 11 # Create encrypted file ansible-vault create secrets.yml # Edit encrypted file ansible-vault edit secrets.yml # Use in playbook ansible-playbook site.yml --ask-vault-pass # Or with password file ansible-playbook site.yml --vault-password-file ~/.vault_pass 1 2 3 4 5 6 7 8 # Encrypted variables file # group_vars/all/vault.yml vault_db_password: !vault | $ANSIBLE_VAULT;1.1;AES256 ... # Reference in playbook db_password: "{{ vault_db_password }}" No Secrets in Logs 1 2 3 4 5 - name: Set database password mysql_user: name: app password: "{{ db_password }}" no_log: true Performance Gather Facts Selectively 1 2 3 4 5 6 7 8 9 10 11 - hosts: webservers gather_facts: false tasks: - name: Quick task without facts ping: # Or gather specific facts - hosts: webservers gather_subset: - network - hardware Async for Long Tasks 1 2 3 4 5 6 7 8 9 10 11 12 13 - name: Run long backup command: /opt/scripts/backup.sh async: 3600 # 1 hour timeout poll: 0 # Don't wait register: backup_job - name: Check backup status async_status: jid: "{{ backup_job.ansible_job_id }}" register: job_result until: job_result.finished retries: 60 delay: 60 Limit Concurrent Execution 1 2 3 4 - hosts: webservers serial: 2 # Two hosts at a time # Or percentage: serial: "25%" # Or batches: serial: [1, 5, 10] Testing with Molecule 1 2 3 4 5 6 7 8 9 10 11 12 13 # molecule/default/molecule.yml dependency: name: galaxy driver: name: docker platforms: - name: instance image: ubuntu:22.04 pre_build_image: true provisioner: name: ansible verifier: name: ansible 1 2 3 4 5 6 7 8 9 10 11 # molecule/default/verify.yml - name: Verify hosts: all tasks: - name: Check nginx is running service: name: nginx state: started check_mode: true register: result failed_when: result.changed 1 2 # Run tests molecule test Good Ansible is boring Ansible. No surprises, no side effects, same result every time. When your playbooks are truly idempotent, running them becomes a confidence-builder rather than a risk. ...

Docker Compose bridges the gap between single-container development and full orchestration. These patterns will help you build maintainable, production-ready configurations. Project Structure m ├ ├ ├ ├ ├ ├ └ y ─ ─ ─ ─ ─ ─ ─ p ─ ─ ─ ─ ─ ─ ─ r o d d d d . . s ├ │ ├ │ └ j o o o o e e e ─ ─ ─ e c c c c n n r ─ ─ ─ c k k k k v v v t e e e e . i a └ w └ n ├ └ / r r r r e c p ─ o ─ g ─ ─ - - - - x e p ─ r ─ i ─ ─ c c c c a s k n o o o o m D e D x D n m m m m p o r / o g p p p p l c c c i o o o o e k k k n s s s s e e e x e e e e r r r . . . . . f f f c y o p t i i i o m v r e l l l n l e o s e e e f r d t r . . i y y d m m e l l . y m l # # # # # # B D P T E T a e r e n e s v o s v m e d t i p u r l c v c c o a o e t o n t n r i n m e f r o f e i i n i n ( g d g t c u e u o r s v r v m a e a a m t ( r t r i i a r i i t o u i o a t n t d n b e o e l d - s e ) l s o a d e d ) Base Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 # docker-compose.yml version: "3.8" services: app: build: context: ./services/app dockerfile: Dockerfile environment: - DATABASE_URL=${DATABASE_URL} - REDIS_URL=${REDIS_URL} depends_on: db: condition: service_healthy redis: condition: service_started networks: - backend restart: unless-stopped db: image: postgres:15-alpine environment: POSTGRES_DB: ${DB_NAME} POSTGRES_USER: ${DB_USER} POSTGRES_PASSWORD: ${DB_PASSWORD} volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"] interval: 10s timeout: 5s retries: 5 networks: - backend redis: image: redis:7-alpine volumes: - redis_data:/data networks: - backend networks: backend: driver: bridge volumes: postgres_data: redis_data: Development Overrides 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 # docker-compose.override.yml (auto-loaded with docker-compose up) version: "3.8" services: app: build: target: development volumes: - ./src:/app/src:cached - /app/node_modules ports: - "3000:3000" - "9229:9229" # Debugger environment: - DEBUG=true - LOG_LEVEL=debug command: npm run dev db: ports: - "5432:5432" redis: ports: - "6379:6379" Production Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 # docker-compose.prod.yml version: "3.8" services: app: build: target: production deploy: replicas: 3 resources: limits: cpus: '1' memory: 512M reservations: cpus: '0.5' memory: 256M restart_policy: condition: on-failure delay: 5s max_attempts: 3 environment: - NODE_ENV=production - LOG_LEVEL=info logging: driver: json-file options: max-size: "10m" max-file: "3" nginx: image: nginx:alpine ports: - "80:80" - "443:443" volumes: - ./services/nginx/nginx.conf:/etc/nginx/nginx.conf:ro - ./certs:/etc/nginx/certs:ro depends_on: - app networks: - backend - frontend networks: frontend: driver: bridge Run with: ...

You know git add, commit, push, and pull. That gets you through 90% of daily work. But the remaining 10%—untangling merge conflicts, finding bug introductions, managing multiple branches simultaneously—requires deeper knowledge. Interactive Rebase The most powerful tool for cleaning up history before sharing. 1 2 3 4 5 # Rebase last 5 commits git rebase -i HEAD~5 # Rebase onto main git rebase -i main The editor opens with commits listed oldest-first: ...

curl is the universal language of HTTP. Every API doc includes curl examples. Every debugging session starts with “can you curl it?” If you’re not comfortable with curl, you’re missing the most portable tool in your kit. Basic Requests 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # GET (default) curl https://api.example.com/users # With headers shown curl -i https://api.example.com/users # Headers only curl -I https://api.example.com/users # Silent (no progress bar) curl -s https://api.example.com/users # Follow redirects curl -L https://example.com/redirect # Verbose (debug mode) curl -v https://api.example.com/users HTTP Methods 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # POST curl -X POST https://api.example.com/users # PUT curl -X PUT https://api.example.com/users/1 # PATCH curl -X PATCH https://api.example.com/users/1 # DELETE curl -X DELETE https://api.example.com/users/1 # HEAD (headers only, like -I) curl -X HEAD https://api.example.com/users Sending Data Form Data 1 2 3 4 5 6 7 # URL-encoded form curl -X POST https://api.example.com/login \ -d "username=admin&password=secret" # From file curl -X POST https://api.example.com/login \ -d @credentials.txt JSON Data 1 2 3 4 5 6 7 8 9 10 11 12 # Inline JSON curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d '{"name":"Alice","email":"alice@example.com"}' # From file curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d @user.json # Using --json (curl 7.82+) curl --json '{"name":"Alice"}' https://api.example.com/users File Upload 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Single file curl -X POST https://api.example.com/upload \ -F "file=@document.pdf" # Multiple files curl -X POST https://api.example.com/upload \ -F "file1=@doc1.pdf" \ -F "file2=@doc2.pdf" # File with custom filename curl -X POST https://api.example.com/upload \ -F "file=@localname.pdf;filename=remote.pdf" # File with content type curl -X POST https://api.example.com/upload \ -F "file=@image.png;type=image/png" # Mixed form data and files curl -X POST https://api.example.com/upload \ -F "title=My Document" \ -F "file=@document.pdf" Headers 1 2 3 4 5 6 7 8 9 10 11 12 13 # Custom header curl -H "X-Custom-Header: value" https://api.example.com # Multiple headers curl -H "Accept: application/json" \ -H "X-API-Version: 2" \ https://api.example.com # User agent curl -A "MyApp/1.0" https://api.example.com # Referer curl -e "https://example.com" https://api.example.com Authentication Basic Auth 1 2 3 4 5 6 7 8 # Username and password curl -u username:password https://api.example.com # Prompt for password curl -u username https://api.example.com # In URL (not recommended) curl https://username:password@api.example.com Bearer Token 1 curl -H "Authorization: Bearer YOUR_TOKEN" https://api.example.com API Key 1 2 3 4 5 # In header curl -H "X-API-Key: YOUR_KEY" https://api.example.com # In query string curl "https://api.example.com?api_key=YOUR_KEY" OAuth 2.0 Flow 1 2 3 4 5 6 7 8 9 # Get access token curl -X POST https://auth.example.com/oauth/token \ -d "grant_type=client_credentials" \ -d "client_id=YOUR_CLIENT_ID" \ -d "client_secret=YOUR_SECRET" # Use token TOKEN="eyJ..." curl -H "Authorization: Bearer $TOKEN" https://api.example.com/resource Digest Auth 1 curl --digest -u username:password https://api.example.com Output Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Save to file curl -o output.html https://example.com # Save with remote filename curl -O https://example.com/file.zip # Save multiple files curl -O https://example.com/file1.zip -O https://example.com/file2.zip # Append to file curl https://example.com >> output.txt # Write headers to file curl -D headers.txt https://example.com # Output to stdout and file curl https://example.com | tee output.html Timeouts and Retries 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Connection timeout (seconds) curl --connect-timeout 5 https://api.example.com # Max time for entire operation curl -m 30 https://api.example.com # Retry on failure curl --retry 3 https://api.example.com # Retry with delay curl --retry 3 --retry-delay 5 https://api.example.com # Retry on specific errors curl --retry 3 --retry-all-errors https://api.example.com SSL/TLS 1 2 3 4 5 6 7 8 9 10 11 12 # Skip certificate verification (insecure!) curl -k https://self-signed.example.com # Use specific CA certificate curl --cacert /path/to/ca.crt https://api.example.com # Client certificate curl --cert client.crt --key client.key https://api.example.com # Force TLS version curl --tlsv1.2 https://api.example.com curl --tlsv1.3 https://api.example.com Proxy 1 2 3 4 5 6 7 8 9 10 11 # HTTP proxy curl -x http://proxy:8080 https://api.example.com # SOCKS5 proxy curl --socks5 localhost:1080 https://api.example.com # Proxy with auth curl -x http://user:pass@proxy:8080 https://api.example.com # No proxy for specific hosts curl --noproxy "localhost,*.internal" https://api.example.com Cookies 1 2 3 4 5 6 7 8 9 10 11 # Send cookie curl -b "session=abc123" https://api.example.com # Send cookies from file curl -b cookies.txt https://api.example.com # Save cookies to file curl -c cookies.txt https://api.example.com/login # Full session (save and send) curl -b cookies.txt -c cookies.txt https://api.example.com Response Inspection 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # HTTP status code only curl -s -o /dev/null -w "%{http_code}" https://api.example.com # Response time curl -s -o /dev/null -w "%{time_total}s" https://api.example.com # Detailed timing curl -s -o /dev/null -w " DNS: %{time_namelookup}s Connect: %{time_connect}s TLS: %{time_appconnect}s Start: %{time_starttransfer}s Total: %{time_total}s Size: %{size_download} bytes Speed: %{speed_download} bytes/sec " https://api.example.com # Content type curl -s -o /dev/null -w "%{content_type}" https://api.example.com Scripting Patterns Health Check 1 2 3 4 5 6 7 8 9 10 11 #!/bin/bash URL="https://api.example.com/health" STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$URL") if [ "$STATUS" -eq 200 ]; then echo "OK" exit 0 else echo "FAIL: HTTP $STATUS" exit 1 fi API Wrapper 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #!/bin/bash API_BASE="https://api.example.com" API_KEY="${API_KEY:?API_KEY required}" api_get() { curl -s -H "Authorization: Bearer $API_KEY" "$API_BASE$1" } api_post() { curl -s -X POST \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$2" \ "$API_BASE$1" } # Usage api_get "/users" | jq '.' api_post "/users" '{"name":"Alice"}' Retry with Backoff 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 #!/bin/bash URL="$1" MAX_RETRIES=5 RETRY_DELAY=1 for i in $(seq 1 $MAX_RETRIES); do RESPONSE=$(curl -s -w "\n%{http_code}" "$URL") STATUS=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') if [ "$STATUS" -eq 200 ]; then echo "$BODY" exit 0 fi echo "Attempt $i failed (HTTP $STATUS), retrying in ${RETRY_DELAY}s..." >&2 sleep $RETRY_DELAY RETRY_DELAY=$((RETRY_DELAY * 2)) done echo "Failed after $MAX_RETRIES attempts" >&2 exit 1 Parallel Requests 1 2 3 4 5 # Using xargs cat urls.txt | xargs -P 10 -I {} curl -s -o /dev/null -w "{}: %{http_code}\n" {} # Using GNU parallel parallel -j 10 curl -s -o /dev/null -w "{}: %{http_code}\n" ::: $(cat urls.txt) Config Files Create ~/.curlrc for defaults: ...

Every API returns JSON. Every config file is JSON. If you’re not fluent in jq, you’re copying data by hand like it’s 1995. The Basics 1 2 3 4 5 6 7 8 9 10 # Pretty print echo '{"name":"test","value":42}' | jq '.' # Extract a field echo '{"name":"test","value":42}' | jq '.name' # "test" # Raw output (no quotes) echo '{"name":"test","value":42}' | jq -r '.name' # test Working with APIs 1 2 3 4 5 6 7 8 # GitHub API curl -s https://api.github.com/users/torvalds | jq '.login, .public_repos' # Extract specific fields curl -s https://api.github.com/repos/stedolan/jq | jq '{name, stars: .stargazers_count, language}' # AWS CLI (already outputs JSON) aws ec2 describe-instances | jq '.Reservations[].Instances[] | {id: .InstanceId, state: .State.Name}' Array Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # Sample data DATA='[{"name":"alice","age":30},{"name":"bob","age":25},{"name":"carol","age":35}]' # First element echo $DATA | jq '.[0]' # Last element echo $DATA | jq '.[-1]' # Slice echo $DATA | jq '.[0:2]' # All names echo $DATA | jq '.[].name' # Array of names echo $DATA | jq '[.[].name]' # Length echo $DATA | jq 'length' Filtering 1 2 3 4 5 6 7 8 9 10 11 # Select by condition echo $DATA | jq '.[] | select(.age > 28)' # Multiple conditions echo $DATA | jq '.[] | select(.age > 25 and .name != "carol")' # Contains echo '[{"tags":["web","api"]},{"tags":["cli"]}]' | jq '.[] | select(.tags | contains(["api"]))' # Has key echo '{"a":1,"b":null}' | jq 'has("a"), has("c")' Transformation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Add/modify fields echo '{"name":"test"}' | jq '. + {status: "active", count: 0}' # Update existing field echo '{"count":5}' | jq '.count += 1' # Delete field echo '{"a":1,"b":2,"c":3}' | jq 'del(.b)' # Rename key echo '{"old_name":"value"}' | jq '{new_name: .old_name}' # Map over array echo '[1,2,3,4,5]' | jq 'map(. * 2)' # Map with objects echo $DATA | jq 'map({username: .name, birth_year: (2026 - .age)})' String Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Concatenation echo '{"first":"John","last":"Doe"}' | jq '.first + " " + .last' # String interpolation echo '{"name":"test","ver":"1.0"}' | jq '"\(.name)-\(.ver).tar.gz"' # Split echo '{"path":"/usr/local/bin"}' | jq '.path | split("/")' # Join echo '["a","b","c"]' | jq 'join(",")' # Upper/lower echo '"Hello World"' | jq 'ascii_downcase' echo '"Hello World"' | jq 'ascii_upcase' # Test regex echo '{"email":"test@example.com"}' | jq '.email | test("@")' # Replace echo '"hello world"' | jq 'gsub("world"; "jq")' Conditionals 1 2 3 4 5 6 7 8 9 10 11 # If-then-else echo '{"status":200}' | jq 'if .status == 200 then "ok" else "error" end' # Alternative operator (default value) echo '{"a":1}' | jq '.b // "default"' # Null handling echo '{"a":null}' | jq '.a // "was null"' # Error handling echo '{}' | jq '.missing.nested // "not found"' Grouping and Aggregation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 LOGS='[ {"level":"error","msg":"failed"}, {"level":"info","msg":"started"}, {"level":"error","msg":"timeout"}, {"level":"info","msg":"completed"} ]' # Group by field echo $LOGS | jq 'group_by(.level)' # Count per group echo $LOGS | jq 'group_by(.level) | map({level: .[0].level, count: length})' # Unique values echo $LOGS | jq '[.[].level] | unique' # Sort echo $DATA | jq 'sort_by(.age)' # Reverse sort echo $DATA | jq 'sort_by(.age) | reverse' # Min/max echo '[5,2,8,1,9]' | jq 'min, max' # Sum echo '[1,2,3,4,5]' | jq 'add' # Average echo '[1,2,3,4,5]' | jq 'add / length' Constructing Output 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Build new object curl -s https://api.github.com/users/torvalds | jq '{ username: .login, repos: .public_repos, profile: .html_url }' # Build array echo '{"users":[{"name":"a"},{"name":"b"}]}' | jq '[.users[].name]' # Multiple outputs to array echo '{"a":1,"b":2}' | jq '[.a, .b, .a + .b]' # Key-value pairs echo '{"a":1,"b":2}' | jq 'to_entries' # [{"key":"a","value":1},{"key":"b","value":2}] # Back to object echo '[{"key":"a","value":1}]' | jq 'from_entries' # Transform keys echo '{"old_a":1,"old_b":2}' | jq 'with_entries(.key |= ltrimstr("old_"))' Real-World Examples Parse AWS Instance List 1 2 3 4 5 aws ec2 describe-instances | jq -r ' .Reservations[].Instances[] | [.InstanceId, .State.Name, (.Tags[]? | select(.Key=="Name") | .Value) // "unnamed"] | @tsv ' Filter Docker Containers 1 2 3 4 5 6 docker inspect $(docker ps -q) | jq '.[] | { name: .Name, image: .Config.Image, status: .State.Status, ip: .NetworkSettings.IPAddress }' Process Log Files 1 2 3 4 5 6 7 # Count errors by type cat app.log | jq -s 'group_by(.error_type) | map({type: .[0].error_type, count: length}) | sort_by(.count) | reverse' # Extract errors from last hour cat app.log | jq --arg cutoff "$(date -d '1 hour ago' -Iseconds)" ' select(.timestamp > $cutoff and .level == "error") ' Transform Config Files 1 2 3 4 5 6 7 8 # Merge configs jq -s '.[0] * .[1]' base.json override.json # Update nested value jq '.database.host = "newhost.example.com"' config.json # Add to array jq '.allowed_ips += ["10.0.0.5"]' config.json Generate Reports 1 2 3 4 5 6 # Kubernetes pod status kubectl get pods -o json | jq -r ' .items[] | [.metadata.name, .status.phase, (.status.containerStatuses[0].restartCount // 0)] | @tsv ' | column -t Useful Flags 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 # Compact output (no pretty print) jq -c '.' # Raw output (no quotes on strings) jq -r '.name' # Raw input (treat input as string, not JSON) jq -R 'split(",")' # Slurp (read all inputs into array) cat *.json | jq -s '.' # Pass variable jq --arg name "test" '.name = $name' # Pass JSON variable jq --argjson count 42 '.count = $count' # Read from file jq --slurpfile users users.json '.users = $users' # Exit with error if output is null/false jq -e '.important_field' && echo "exists" # Sort keys in output jq -S '.' Output Formats 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Tab-separated echo $DATA | jq -r '.[] | [.name, .age] | @tsv' # CSV echo $DATA | jq -r '.[] | [.name, .age] | @csv' # URI encoding echo '{"q":"hello world"}' | jq -r '.q | @uri' # Base64 echo '{"data":"secret"}' | jq -r '.data | @base64' # Shell-safe echo '{"cmd":"echo hello"}' | jq -r '.cmd | @sh' Debugging 1 2 3 4 5 6 7 8 9 10 11 # Show type echo '{"a":[1,2,3]}' | jq '.a | type' # Show keys echo '{"a":1,"b":2}' | jq 'keys' # Debug output (shows intermediate values) echo '{"x":{"y":{"z":1}}}' | jq '.x | debug | .y | debug | .z' # Path to value echo '{"a":{"b":{"c":1}}}' | jq 'path(.. | select(. == 1))' Quick Reference 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 # Identity . # Field access .field .field.nested # Array access .[0] .[-1] .[2:5] # Iterate array .[] # Pipe .[] | .name # Collect into array [.[] | .name] # Object construction {newkey: .oldkey} # Conditionals if COND then A else B end VALUE // DEFAULT # Comparison ==, !=, <, >, <=, >= and, or, not # Array functions map(f), select(f), sort_by(f), group_by(f), unique, length, first, last, nth(n), flatten, reverse, contains(x), inside(x), add, min, max # String functions split(s), join(s), test(re), match(re), gsub(re;s), ascii_downcase, ascii_upcase, ltrimstr(s), rtrimstr(s), startswith(s), endswith(s) # Object functions keys, values, has(k), in(o), to_entries, from_entries, with_entries(f) # Type functions type, isnumber, isstring, isnull, isboolean, isarray, isobject jq turns JSON from a data format into a query language. Once you internalize the pipe-and-filter model, you’ll wonder how you ever survived without it. ...

Cron has run scheduled tasks since 1975. It works, but systemd timers offer significant advantages: integrated logging, dependency management, randomized delays, and calendar-based scheduling that actually makes sense. Why Switch from Cron? Logging: Timer output goes to journald. No more digging through mail or custom log files. Dependencies: Wait for network, mounts, or other services before running. Accuracy: Monotonic timers don’t drift. Calendar timers handle DST correctly. Visibility: systemctl list-timers shows all scheduled jobs and when they’ll run next. ...

The third factor of the 12-Factor App methodology states: “Store config in the environment.” Simple advice that’s surprisingly easy to get wrong. The Core Principle Configuration that varies between environments (dev, staging, production) should come from environment variables, not code. This includes: Database connection strings API keys and secrets Feature flags Service URLs Port numbers Log levels What stays in code: application logic, default behaviors, anything that doesn’t change between deploys. ...

Make was designed for compiling C programs in 1976. Nearly 50 years later, it’s still one of the most practical automation tools available—not for its original purpose, but as a universal task runner. Why Make in 2026? It’s already installed. Every Unix system has make. No npm install, no pip, no version managers. It’s declarative. Define what you want, not how to get there (with dependencies handled automatically). It’s documented. make help can list all your targets. The Makefile itself is documentation. ...

If you’re still typing ssh -i ~/.ssh/my-key.pem -p 2222 admin@192.168.1.50 every time you connect, you’re doing it wrong. The SSH config file is one of the most underutilized productivity tools in a developer’s arsenal. The Basics: ~/.ssh/config Create or edit ~/.ssh/config: 1 2 3 4 5 Host dev HostName dev.example.com User deploy IdentityFile ~/.ssh/deploy_key Port 22 Now you just type ssh dev. That’s it. Host Patterns Wildcards let you apply settings to multiple hosts: ...