DevOps

Understanding processes is fundamental to Linux troubleshooting. These tools and techniques will help you find what’s running, what’s stuck, and what needs to die. Viewing Processes ps - Process Snapshot 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # All processes (BSD style) ps aux # All processes (Unix style) ps -ef # Process tree ps auxf # Specific columns ps -eo pid,ppid,user,%cpu,%mem,stat,cmd # Find specific process ps aux | grep nginx # By exact name (no grep needed) ps -C nginx # By user ps -u www-data Understanding ps Output U r w S o w E o w R t - d a t a 1 P 2 I 3 D 1 4 % C 0 2 P . . U 0 5 % M 0 1 E . . M 1 2 1 4 6 5 9 6 V 9 7 S 3 8 Z 6 9 1 9 3 8 R 2 7 S 5 6 S 6 5 T ? ? T Y S S S T s l A T S F 1 T e 0 A b : R 2 0 T 4 0 T 0 5 I : : M 0 2 E 3 3 C / n O s g M b i M i n A n x N / : D i n w i o t r k e r PID: Process ID %CPU: CPU usage %MEM: Memory usage VSZ: Virtual memory size RSS: Resident set size (actual RAM) STAT: Process state TIME: CPU time consumed Process States (STAT) R S D Z T N s l R S S Z S H L S M F u l l o t i o e u o n e e m o g w s l r n e e b p h s t e i p p i p p i i g n i i e e p r o - r g n n d r i n t o g g i o h u o r l r n ( ( r i e e d i u i t a a n n t y d d p t i y e e r e n r d o r t c r e e u r s p r s t u i p b t l i e b ) l e , u s u a l l y I / O ) top - Real-time View 1 2 3 4 5 6 7 8 9 10 11 # Basic top # Sort by memory top -o %MEM # Specific user top -u www-data # Batch mode (for scripts) top -b -n 1 Inside top: ...

Terraform state is the source of truth for your infrastructure. Mismanage it, and you’ll have drift, conflicts, and 3 AM incidents. These patterns keep state safe and teams productive. Why State Matters Terraform state maps your configuration to real resources. Without it, Terraform can’t: Know what already exists Calculate what needs to change Detect drift from desired state Default local state (terraform.tfstate) breaks down quickly: Can’t collaborate (who has the latest?) No locking (concurrent runs = corruption) No history (oops, we deleted production) Remote Backend: S3 + DynamoDB The standard pattern for AWS teams: ...

Nginx powers a significant portion of the internet. These configurations will help you move from default installs to production-ready setups. Basic Structure 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 # /etc/nginx/nginx.conf user www-data; worker_processes auto; pid /run/nginx.pid; events { worker_connections 1024; multi_accept on; } http { include /etc/nginx/mime.types; default_type application/octet-stream; # Logging access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; # Performance sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; # Gzip gzip on; gzip_types text/plain text/css application/json application/javascript; # Virtual hosts include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } Basic Server Block 1 2 3 4 5 6 7 8 9 10 11 12 13 # /etc/nginx/sites-available/example.com server { listen 80; listen [::]:80; server_name example.com www.example.com; root /var/www/example.com; index index.html; location / { try_files $uri $uri/ =404; } } SSL/TLS Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name example.com; # Certificates ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # Modern SSL configuration ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256; ssl_prefer_server_ciphers off; # SSL session caching ssl_session_cache shared:SSL:10m; ssl_session_timeout 1d; ssl_session_tickets off; # OCSP stapling ssl_stapling on; ssl_stapling_verify on; resolver 8.8.8.8 8.8.4.4 valid=300s; root /var/www/example.com; } # Redirect HTTP to HTTPS server { listen 80; listen [::]:80; server_name example.com www.example.com; return 301 https://$server_name$request_uri; } Reverse Proxy Basic Proxy 1 2 3 4 5 6 7 8 9 10 11 12 13 server { listen 80; server_name api.example.com; location / { proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } WebSocket Support 1 2 3 4 5 6 7 8 location /ws { proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_read_timeout 86400; } Proxy with Buffering Control 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 location /api { proxy_pass http://backend; # Disable buffering for streaming proxy_buffering off; # Or tune buffer sizes proxy_buffer_size 4k; proxy_buffers 8 4k; proxy_busy_buffers_size 8k; # Timeouts proxy_connect_timeout 60s; proxy_send_timeout 60s; proxy_read_timeout 60s; } Load Balancing 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 upstream backend { # Round robin (default) server 10.0.0.1:3000; server 10.0.0.2:3000; server 10.0.0.3:3000; # Weighted server 10.0.0.1:3000 weight=3; server 10.0.0.2:3000 weight=1; # Backup server server 10.0.0.4:3000 backup; # Health checks server 10.0.0.1:3000 max_fails=3 fail_timeout=30s; } # Alternative algorithms upstream backend_least_conn { least_conn; server 10.0.0.1:3000; server 10.0.0.2:3000; } upstream backend_ip_hash { ip_hash; # Session persistence server 10.0.0.1:3000; server 10.0.0.2:3000; } server { location / { proxy_pass http://backend; } } Caching Proxy Cache 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 http { # Define cache zone proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m use_temp_path=off; server { location / { proxy_pass http://backend; proxy_cache my_cache; proxy_cache_valid 200 60m; proxy_cache_valid 404 1m; proxy_cache_use_stale error timeout updating; # Cache key proxy_cache_key "$scheme$request_method$host$request_uri"; # Add header to show cache status add_header X-Cache-Status $upstream_cache_status; } # Bypass cache for specific requests location /api { proxy_pass http://backend; proxy_cache my_cache; proxy_cache_bypass $http_authorization; proxy_no_cache $http_authorization; } } } Static File Caching 1 2 3 4 5 location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff2)$ { expires 30d; add_header Cache-Control "public, immutable"; access_log off; } Rate Limiting 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 http { # Define rate limit zones limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s; limit_req_zone $binary_remote_addr zone=login_limit:10m rate=1r/s; server { # Apply to API endpoints location /api/ { limit_req zone=api_limit burst=20 nodelay; proxy_pass http://backend; } # Stricter limit for login location /login { limit_req zone=login_limit burst=5; proxy_pass http://backend; } } } Connection Limiting 1 2 3 4 5 6 7 http { limit_conn_zone $binary_remote_addr zone=conn_limit:10m; server { limit_conn conn_limit 10; # Max 10 connections per IP } } Security Headers 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 server { # Security headers add_header X-Frame-Options "SAMEORIGIN" always; add_header X-Content-Type-Options "nosniff" always; add_header X-XSS-Protection "1; mode=block" always; add_header Referrer-Policy "strict-origin-when-cross-origin" always; add_header Content-Security-Policy "default-src 'self';" always; # HSTS (only enable after confirming SSL works) add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; # Hide nginx version server_tokens off; # Prevent access to hidden files location ~ /\. { deny all; } } Access Control IP-based 1 2 3 4 5 6 7 location /admin { allow 10.0.0.0/8; allow 192.168.1.0/24; deny all; proxy_pass http://backend; } Basic Authentication 1 2 3 4 5 6 location /admin { auth_basic "Admin Area"; auth_basic_user_file /etc/nginx/.htpasswd; proxy_pass http://backend; } Create password file: ...

Redis is often introduced as “just a cache,” but it’s a versatile data structure server. These patterns unlock its full potential. Connection Basics 1 2 3 4 5 6 7 8 9 10 11 # Connect redis-cli -h localhost -p 6379 # With password redis-cli -h localhost -p 6379 -a yourpassword # Select database (0-15) SELECT 1 # Check connectivity PING Caching Patterns Basic Cache with TTL 1 2 3 4 5 6 7 8 9 10 11 # Set with expiration (seconds) SET user:123:profile '{"name":"Alice"}' EX 3600 # Set with expiration (milliseconds) SET session:abc123 '{"user_id":123}' PX 86400000 # Set only if not exists SETNX cache:key "value" # Set only if exists (update) SET cache:key "newvalue" XX Cache-Aside Pattern 1 2 3 4 5 6 7 8 9 10 11 12 def get_user(user_id): # Check cache first cached = redis.get(f"user:{user_id}") if cached: return json.loads(cached) # Cache miss - fetch from database user = db.query("SELECT * FROM users WHERE id = %s", user_id) # Store in cache redis.setex(f"user:{user_id}", 3600, json.dumps(user)) return user Write-Through Pattern 1 2 3 4 5 6 def update_user(user_id, data): # Update database db.execute("UPDATE users SET ... WHERE id = %s", user_id) # Update cache immediately redis.setex(f"user:{user_id}", 3600, json.dumps(data)) Cache Stampede Prevention 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 def get_with_lock(key, fetch_func, ttl=3600, lock_ttl=10): value = redis.get(key) if value: return json.loads(value) lock_key = f"lock:{key}" # Try to acquire lock if redis.set(lock_key, "1", nx=True, ex=lock_ttl): try: value = fetch_func() redis.setex(key, ttl, json.dumps(value)) return value finally: redis.delete(lock_key) else: # Another process is fetching, wait and retry time.sleep(0.1) return get_with_lock(key, fetch_func, ttl, lock_ttl) Session Storage 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 import secrets def create_session(user_id, ttl=86400): session_id = secrets.token_urlsafe(32) session_data = { "user_id": user_id, "created_at": time.time() } redis.setex(f"session:{session_id}", ttl, json.dumps(session_data)) return session_id def get_session(session_id): data = redis.get(f"session:{session_id}") return json.loads(data) if data else None def extend_session(session_id, ttl=86400): redis.expire(f"session:{session_id}", ttl) def destroy_session(session_id): redis.delete(f"session:{session_id}") Rate Limiting Fixed Window 1 2 3 4 5 6 7 8 def is_rate_limited(user_id, limit=100, window=60): key = f"ratelimit:{user_id}:{int(time.time() // window)}" current = redis.incr(key) if current == 1: redis.expire(key, window) return current > limit Sliding Window with Sorted Sets 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 def is_rate_limited_sliding(user_id, limit=100, window=60): key = f"ratelimit:{user_id}" now = time.time() window_start = now - window pipe = redis.pipeline() # Remove old entries pipe.zremrangebyscore(key, 0, window_start) # Add current request pipe.zadd(key, {str(now): now}) # Count requests in window pipe.zcard(key) # Set expiration pipe.expire(key, window) results = pipe.execute() request_count = results[2] return request_count > limit Token Bucket 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 def check_token_bucket(user_id, capacity=10, refill_rate=1): key = f"bucket:{user_id}" now = time.time() # Get current state data = redis.hgetall(key) if data: tokens = float(data[b'tokens']) last_update = float(data[b'last_update']) # Refill tokens based on elapsed time elapsed = now - last_update tokens = min(capacity, tokens + elapsed * refill_rate) else: tokens = capacity if tokens >= 1: # Consume a token redis.hset(key, mapping={ 'tokens': tokens - 1, 'last_update': now }) redis.expire(key, int(capacity / refill_rate) + 1) return True return False Queues and Pub/Sub Simple Queue with Lists 1 2 3 4 5 6 7 8 9 10 # Producer def enqueue(queue_name, message): redis.lpush(queue_name, json.dumps(message)) # Consumer (blocking) def dequeue(queue_name, timeout=0): result = redis.brpop(queue_name, timeout) if result: return json.loads(result[1]) return None Reliable Queue with RPOPLPUSH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 def reliable_dequeue(queue_name, processing_queue): # Move item to processing queue atomically item = redis.rpoplpush(queue_name, processing_queue) return json.loads(item) if item else None def ack(processing_queue, item): # Remove from processing queue when done redis.lrem(processing_queue, 1, json.dumps(item)) def requeue_failed(processing_queue, queue_name): # Move failed items back to main queue while True: item = redis.rpoplpush(processing_queue, queue_name) if not item: break Pub/Sub 1 2 3 4 5 6 7 8 9 10 11 12 # Publisher def publish_event(channel, event): redis.publish(channel, json.dumps(event)) # Subscriber def subscribe(channel, callback): pubsub = redis.pubsub() pubsub.subscribe(channel) for message in pubsub.listen(): if message['type'] == 'message': callback(json.loads(message['data'])) Leaderboards with Sorted Sets 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 def add_score(leaderboard, user_id, score): redis.zadd(leaderboard, {user_id: score}) def increment_score(leaderboard, user_id, amount): redis.zincrby(leaderboard, amount, user_id) def get_rank(leaderboard, user_id): # 0-indexed, reverse order (highest first) rank = redis.zrevrank(leaderboard, user_id) return rank + 1 if rank is not None else None def get_top(leaderboard, count=10): return redis.zrevrange(leaderboard, 0, count - 1, withscores=True) def get_around_user(leaderboard, user_id, count=5): rank = redis.zrevrank(leaderboard, user_id) if rank is None: return [] start = max(0, rank - count) end = rank + count return redis.zrevrange(leaderboard, start, end, withscores=True) Distributed Locks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 import uuid class RedisLock: def __init__(self, redis_client, key, ttl=10): self.redis = redis_client self.key = f"lock:{key}" self.ttl = ttl self.token = str(uuid.uuid4()) def acquire(self, blocking=True, timeout=None): start = time.time() while True: if self.redis.set(self.key, self.token, nx=True, ex=self.ttl): return True if not blocking: return False if timeout and (time.time() - start) > timeout: return False time.sleep(0.1) def release(self): # Only release if we own the lock script = """ if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end """ self.redis.eval(script, 1, self.key, self.token) def __enter__(self): self.acquire() return self def __exit__(self, *args): self.release() # Usage with RedisLock(redis, "my-resource"): # Critical section do_work() Counting and Analytics HyperLogLog for Unique Counts 1 2 3 4 5 6 7 8 9 10 11 # Count unique visitors (memory efficient) def track_visitor(page, visitor_id): redis.pfadd(f"visitors:{page}:{date.today()}", visitor_id) def get_unique_visitors(page, date): return redis.pfcount(f"visitors:{page}:{date}") # Merge multiple days def get_weekly_uniques(page): keys = [f"visitors:{page}:{date}" for date in last_7_days()] return redis.pfcount(*keys) Bitmaps for Daily Active Users 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 def mark_active(user_id, date=None): date = date or date.today().isoformat() redis.setbit(f"active:{date}", user_id, 1) def was_active(user_id, date): return redis.getbit(f"active:{date}", user_id) == 1 def count_active(date): return redis.bitcount(f"active:{date}") # Users active on multiple days def active_all_days(dates): keys = [f"active:{d}" for d in dates] result_key = "temp:active_intersection" redis.bitop("AND", result_key, *keys) count = redis.bitcount(result_key) redis.delete(result_key) return count Expiration Strategies 1 2 3 4 5 6 7 8 9 10 11 12 # Set TTL EXPIRE key 3600 EXPIREAT key 1735689600 # Unix timestamp # Check TTL TTL key # Returns -1 if no expiry, -2 if doesn't exist # Remove expiration PERSIST key # Set value and TTL atomically SETEX key 3600 "value" Lazy Expiration Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 def get_with_soft_expire(key, ttl=3600, soft_ttl=300): """ Returns cached value but triggers background refresh if within soft_ttl of expiration. """ pipe = redis.pipeline() pipe.get(key) pipe.ttl(key) value, remaining_ttl = pipe.execute() if value and remaining_ttl < soft_ttl: # Trigger async refresh refresh_cache_async.delay(key) return value Transactions and Lua Scripts Pipeline (Batching) 1 2 3 4 pipe = redis.pipeline() for i in range(1000): pipe.set(f"key:{i}", f"value:{i}") pipe.execute() # Single round trip Transaction with WATCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 def transfer(from_account, to_account, amount): with redis.pipeline() as pipe: while True: try: # Watch for changes pipe.watch(from_account, to_account) from_balance = int(pipe.get(from_account) or 0) if from_balance < amount: pipe.unwatch() return False # Start transaction pipe.multi() pipe.decrby(from_account, amount) pipe.incrby(to_account, amount) pipe.execute() return True except redis.WatchError: # Retry if watched keys changed continue Lua Script (Atomic Operations) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Rate limiter as Lua script RATE_LIMIT_SCRIPT = """ local key = KEYS[1] local limit = tonumber(ARGV[1]) local window = tonumber(ARGV[2]) local current = redis.call('INCR', key) if current == 1 then redis.call('EXPIRE', key, window) end if current > limit then return 0 else return 1 end """ rate_limit = redis.register_script(RATE_LIMIT_SCRIPT) def check_rate_limit(user_id, limit=100, window=60): key = f"ratelimit:{user_id}:{int(time.time() // window)}" return rate_limit(keys=[key], args=[limit, window]) == 1 Monitoring 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Real-time commands MONITOR # Stats INFO INFO memory INFO stats # Slow queries SLOWLOG GET 10 # Connected clients CLIENT LIST # Memory usage for a key MEMORY USAGE mykey Redis excels when you match the right data structure to your problem. Lists for queues, sorted sets for leaderboards, HyperLogLog for counting uniques—each has its sweet spot. ...

PostgreSQL is fast out of the box. But “fast enough for development” and “fast enough for production” are different conversations. These techniques will help you find and fix performance bottlenecks. Finding Slow Queries Enable Query Logging 1 2 3 4 5 6 -- Log queries slower than 500ms ALTER SYSTEM SET log_min_duration_statement = '500ms'; SELECT pg_reload_conf(); -- Check current setting SHOW log_min_duration_statement; pg_stat_statements Extension The most valuable performance tool: ...

Ansible’s simplicity is deceptive. Anyone can write a playbook that works once. Writing playbooks that work reliably, repeatedly, and maintainably requires discipline and patterns. Project Structure a ├ ├ │ │ │ │ │ │ │ │ ├ │ │ │ ├ │ │ │ └ n ─ ─ ─ ─ ─ s ─ ─ ─ ─ ─ i b a i ├ │ │ │ │ └ p ├ ├ └ r ├ ├ └ f └ l n n ─ ─ l ─ ─ ─ o ─ ─ ─ i ─ e s v ─ ─ a ─ ─ ─ l ─ ─ ─ l ─ / i e y e e b n p ├ └ s ├ └ b s w d s c n p s s l t r ─ ─ t ─ ─ o i e a / o g o / c e o o ─ ─ a ─ ─ o t b t m i s r . r d g k e s a m n t i c y u h g ├ └ i h g s . e b o x g p f / c o r ─ ─ n o r y r a n r t g t s o ─ ─ g s m v s e s i t u / t u l e e s / o s p a w s p r s q n . _ l e . _ s . l y v l b y v . y / m a . s m a y m l r y e l r m l s m r s l / l v / e r s . y m l Inventory Best Practices 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # inventory/production/hosts.yml all: children: webservers: hosts: web1.example.com: web2.example.com: vars: http_port: 80 databases: hosts: db1.example.com: postgres_role: primary db2.example.com: postgres_role: replica loadbalancers: hosts: lb1.example.com: vars: ansible_user: deploy ansible_python_interpreter: /usr/bin/python3 Role Structure r ├ │ ├ │ ├ │ │ │ ├ │ ├ │ ├ │ ├ │ └ o ─ ─ ─ ─ ─ ─ ─ ─ l ─ ─ ─ ─ ─ ─ ─ ─ e s d └ v └ t ├ ├ └ h └ t └ f └ m └ m └ / e ─ a ─ a ─ ─ ─ a ─ e ─ i ─ e ─ o ─ n f ─ r ─ s ─ ─ ─ n ─ m ─ l ─ t ─ l ─ g a s k d p e a e i u m / m s m i c l m l n s s / m c d n l a a / a n o e a a g / s a u e x t i i i s n r i t i l i l f / s n n n t f s n e n / n e a / . . . a i / . s x . / u y y y l g y / . y l m m m l u m c m t l l l . r l o l / y e n m . f l y . m j l 2 # # # # # # D R E S D T e o n e e e f l t r p s a e r v e t u y i n i l v c d n t a p e e g r o n v i i r c a a n e i r b t s e i l t s a e a b s r l t e ( s h h i a ( g n l h d o e l w r e e r s p s t r i p o r r i i o t r y i ) t y ) Task Patterns Always Name Tasks 1 2 3 4 5 6 7 8 9 10 # Bad - apt: name: nginx state: present # Good - name: Install nginx apt: name: nginx state: present Use Block for Related Tasks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 - name: Configure SSL block: - name: Copy SSL certificate copy: src: "{{ ssl_cert }}" dest: /etc/ssl/certs/ mode: '0644' - name: Copy SSL private key copy: src: "{{ ssl_key }}" dest: /etc/ssl/private/ mode: '0600' - name: Enable SSL site file: src: /etc/nginx/sites-available/ssl.conf dest: /etc/nginx/sites-enabled/ssl.conf state: link notify: Reload nginx when: ssl_enabled | bool Error Handling 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 - name: Deploy application block: - name: Pull latest code git: repo: "{{ app_repo }}" dest: "{{ app_path }}" version: "{{ app_version }}" register: git_result - name: Run migrations command: ./manage.py migrate args: chdir: "{{ app_path }}" when: git_result.changed rescue: - name: Rollback to previous version git: repo: "{{ app_repo }}" dest: "{{ app_path }}" version: "{{ previous_version }}" - name: Notify failure slack: token: "{{ slack_token }}" msg: "Deploy failed on {{ inventory_hostname }}" always: - name: Clean up temp files file: path: /tmp/deploy state: absent Variables Variable Precedence (use intentionally) 1 2 3 4 5 6 7 8 9 10 11 12 13 # defaults/main.yml - Easily overridden defaults nginx_worker_processes: auto nginx_worker_connections: 1024 # vars/main.yml - Role-specific constants nginx_user: www-data nginx_conf_path: /etc/nginx/nginx.conf # group_vars/webservers.yml - Group-specific nginx_worker_connections: 4096 # host_vars/web1.yml - Host-specific nginx_worker_processes: 4 Variable Validation 1 2 3 4 5 6 7 8 - name: Validate required variables assert: that: - app_version is defined - app_version | length > 0 - db_password is defined fail_msg: "Required variables are not set" success_msg: "All required variables present" Default Values 1 2 3 4 5 6 7 8 - name: Set configuration template: src: config.j2 dest: /etc/app/config.yml vars: max_connections: "{{ app_max_connections | default(100) }}" timeout: "{{ app_timeout | default(30) }}" debug: "{{ app_debug | default(false) | bool }}" Handlers 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # handlers/main.yml - name: Reload nginx service: name: nginx state: reloaded listen: "reload web server" - name: Restart nginx service: name: nginx state: restarted listen: "restart web server" # In tasks - name: Update nginx config template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf notify: "reload web server" Flush Handlers When Needed 1 2 3 4 5 6 7 8 9 10 11 12 13 14 - name: Install nginx apt: name: nginx state: present notify: Start nginx - name: Flush handlers meta: flush_handlers - name: Configure nginx template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf # nginx is now guaranteed to be running Conditionals Clean Conditionals 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Use bool filter for string booleans - name: Enable debug mode template: src: debug.conf.j2 dest: /etc/app/debug.conf when: debug_mode | bool # Check if variable is defined and not empty - name: Set custom config copy: content: "{{ custom_config }}" dest: /etc/app/custom.conf when: - custom_config is defined - custom_config | length > 0 # Multiple conditions - name: Deploy to production include_tasks: deploy.yml when: - env == 'production' - deploy_enabled | bool - inventory_hostname in groups['webservers'] Loops Modern Loop Syntax 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 # Simple loop - name: Install packages apt: name: "{{ item }}" state: present loop: - nginx - python3 - htop # Better: Install all at once - name: Install packages apt: name: - nginx - python3 - htop state: present # Loop with index - name: Create users user: name: "{{ item.name }}" uid: "{{ 1000 + index }}" groups: "{{ item.groups }}" loop: "{{ users }}" loop_control: index_var: index label: "{{ item.name }}" # Cleaner output # Dict loop - name: Configure services template: src: "{{ item.key }}.conf.j2" dest: "/etc/{{ item.key }}/config.conf" loop: "{{ services | dict2items }}" when: item.value.enabled | bool Templates Jinja2 Best Practices 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 {# templates/nginx.conf.j2 #} # Managed by Ansible - DO NOT EDIT # Last updated: {{ ansible_date_time.iso8601 }} # Host: {{ inventory_hostname }} worker_processes {{ nginx_worker_processes }}; events { worker_connections {{ nginx_worker_connections }}; } http { {% for server in nginx_servers %} server { listen {{ server.port | default(80) }}; server_name {{ server.name }}; {% if server.ssl | default(false) %} ssl_certificate {{ server.ssl_cert }}; ssl_certificate_key {{ server.ssl_key }}; {% endif %} {% for location in server.locations | default([]) %} location {{ location.path }} { {{ location.directive }}; } {% endfor %} } {% endfor %} } Template Validation 1 2 3 4 5 6 - name: Generate nginx config template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf validate: nginx -t -c %s notify: Reload nginx Idempotency Patterns Check Mode Support 1 2 3 4 5 6 7 8 9 10 11 - name: Get current version command: cat /opt/app/VERSION register: current_version changed_when: false check_mode: false # Always run, even in check mode - name: Deploy new version unarchive: src: "app-{{ target_version }}.tar.gz" dest: /opt/app/ when: current_version.stdout != target_version Custom Changed Conditions 1 2 3 4 5 - name: Run database migration command: ./manage.py migrate --check register: migration_check changed_when: "'No migrations to apply' not in migration_check.stdout" failed_when: migration_check.rc not in [0, 1] Avoid Command When Possible 1 2 3 4 5 6 7 8 9 10 # Bad - not idempotent - name: Create directory command: mkdir -p /opt/app # Good - idempotent - name: Create directory file: path: /opt/app state: directory mode: '0755' Secrets Management Ansible Vault 1 2 3 4 5 6 7 8 9 10 11 # Create encrypted file ansible-vault create secrets.yml # Edit encrypted file ansible-vault edit secrets.yml # Use in playbook ansible-playbook site.yml --ask-vault-pass # Or with password file ansible-playbook site.yml --vault-password-file ~/.vault_pass 1 2 3 4 5 6 7 8 # Encrypted variables file # group_vars/all/vault.yml vault_db_password: !vault | $ANSIBLE_VAULT;1.1;AES256 ... # Reference in playbook db_password: "{{ vault_db_password }}" No Secrets in Logs 1 2 3 4 5 - name: Set database password mysql_user: name: app password: "{{ db_password }}" no_log: true Performance Gather Facts Selectively 1 2 3 4 5 6 7 8 9 10 11 - hosts: webservers gather_facts: false tasks: - name: Quick task without facts ping: # Or gather specific facts - hosts: webservers gather_subset: - network - hardware Async for Long Tasks 1 2 3 4 5 6 7 8 9 10 11 12 13 - name: Run long backup command: /opt/scripts/backup.sh async: 3600 # 1 hour timeout poll: 0 # Don't wait register: backup_job - name: Check backup status async_status: jid: "{{ backup_job.ansible_job_id }}" register: job_result until: job_result.finished retries: 60 delay: 60 Limit Concurrent Execution 1 2 3 4 - hosts: webservers serial: 2 # Two hosts at a time # Or percentage: serial: "25%" # Or batches: serial: [1, 5, 10] Testing with Molecule 1 2 3 4 5 6 7 8 9 10 11 12 13 # molecule/default/molecule.yml dependency: name: galaxy driver: name: docker platforms: - name: instance image: ubuntu:22.04 pre_build_image: true provisioner: name: ansible verifier: name: ansible 1 2 3 4 5 6 7 8 9 10 11 # molecule/default/verify.yml - name: Verify hosts: all tasks: - name: Check nginx is running service: name: nginx state: started check_mode: true register: result failed_when: result.changed 1 2 # Run tests molecule test Good Ansible is boring Ansible. No surprises, no side effects, same result every time. When your playbooks are truly idempotent, running them becomes a confidence-builder rather than a risk. ...

Bash scripts have a reputation for being fragile, unreadable hacks. They don’t have to be. These patterns will help you write scripts that are maintainable, debuggable, and safe to run in production. The Essentials: Start Every Script Right 1 2 3 4 #!/usr/bin/env bash set -euo pipefail IFS=$'\n\t' What these do: set -e — Exit on any error set -u — Error on undefined variables set -o pipefail — Catch errors in pipelines IFS=$'\n\t' — Safer word splitting (no spaces) Script Template 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 #!/usr/bin/env bash set -euo pipefail # Script metadata readonly SCRIPT_NAME="$(basename "$0")" readonly SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" readonly SCRIPT_VERSION="1.0.0" # Default values VERBOSE=false DRY_RUN=false CONFIG_FILE="" # Colors (if terminal supports them) if [[ -t 1 ]]; then readonly RED='\033[0;31m' readonly GREEN='\033[0;32m' readonly YELLOW='\033[0;33m' readonly NC='\033[0m' # No Color else readonly RED='' GREEN='' YELLOW='' NC='' fi # Logging functions log_info() { echo -e "${GREEN}[INFO]${NC} $*"; } log_warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; } log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; } log_debug() { [[ "$VERBOSE" == true ]] && echo -e "[DEBUG] $*" || true; } # Cleanup function cleanup() { local exit_code=$? # Add cleanup tasks here log_debug "Cleaning up..." exit $exit_code } trap cleanup EXIT # Usage usage() { cat <<EOF Usage: $SCRIPT_NAME [OPTIONS] <argument> Description of what this script does. Options: -h, --help Show this help message -v, --verbose Enable verbose output -n, --dry-run Show what would be done -c, --config FILE Path to config file --version Show version Examples: $SCRIPT_NAME -v input.txt $SCRIPT_NAME --config /etc/myapp.conf data/ EOF } # Parse arguments parse_args() { while [[ $# -gt 0 ]]; do case $1 in -h|--help) usage exit 0 ;; -v|--verbose) VERBOSE=true shift ;; -n|--dry-run) DRY_RUN=true shift ;; -c|--config) CONFIG_FILE="$2" shift 2 ;; --version) echo "$SCRIPT_NAME version $SCRIPT_VERSION" exit 0 ;; -*) log_error "Unknown option: $1" usage exit 1 ;; *) ARGS+=("$1") shift ;; esac done } # Validation validate() { if [[ ${#ARGS[@]} -eq 0 ]]; then log_error "Missing required argument" usage exit 1 fi if [[ -n "$CONFIG_FILE" && ! -f "$CONFIG_FILE" ]]; then log_error "Config file not found: $CONFIG_FILE" exit 1 fi } # Main logic main() { log_info "Starting $SCRIPT_NAME" log_debug "Arguments: ${ARGS[*]}" for arg in "${ARGS[@]}"; do if [[ "$DRY_RUN" == true ]]; then log_info "[DRY RUN] Would process: $arg" else log_info "Processing: $arg" # Actual work here fi done log_info "Done" } # Entry point ARGS=() parse_args "$@" validate main Error Handling Check Command Existence 1 2 3 4 5 6 7 8 9 10 require_command() { if ! command -v "$1" &> /dev/null; then log_error "Required command not found: $1" exit 1 fi } require_command curl require_command jq require_command docker Retry Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 retry() { local max_attempts=$1 local delay=$2 shift 2 local cmd=("$@") local attempt=1 while (( attempt <= max_attempts )); do if "${cmd[@]}"; then return 0 fi log_warn "Attempt $attempt/$max_attempts failed, retrying in ${delay}s..." sleep "$delay" ((attempt++)) done log_error "Command failed after $max_attempts attempts: ${cmd[*]}" return 1 } # Usage retry 3 5 curl -sf https://api.example.com/health Timeout Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 with_timeout() { local timeout=$1 shift timeout "$timeout" "$@" || { local exit_code=$? if [[ $exit_code -eq 124 ]]; then log_error "Command timed out after ${timeout}s" fi return $exit_code } } # Usage with_timeout 30 curl -s https://api.example.com Safe File Operations Temporary Files 1 2 3 4 5 6 7 # Create temp file that's automatically cleaned up TEMP_FILE=$(mktemp) trap 'rm -f "$TEMP_FILE"' EXIT # Or temp directory TEMP_DIR=$(mktemp -d) trap 'rm -rf "$TEMP_DIR"' EXIT Safe Write (Atomic) 1 2 3 4 5 6 7 8 9 10 11 12 safe_write() { local target=$1 local content=$2 local temp_file temp_file=$(mktemp) echo "$content" > "$temp_file" mv "$temp_file" "$target" } # Usage safe_write /etc/myapp/config.json '{"key": "value"}' Backup Before Modify 1 2 3 4 5 6 7 8 9 backup_file() { local file=$1 local backup="${file}.bak.$(date +%Y%m%d_%H%M%S)" if [[ -f "$file" ]]; then cp "$file" "$backup" log_info "Backup created: $backup" fi } Input Validation File Checks 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 validate_file() { local file=$1 [[ -e "$file" ]] || { log_error "File not found: $file"; return 1; } [[ -f "$file" ]] || { log_error "Not a regular file: $file"; return 1; } [[ -r "$file" ]] || { log_error "File not readable: $file"; return 1; } } validate_directory() { local dir=$1 [[ -e "$dir" ]] || { log_error "Directory not found: $dir"; return 1; } [[ -d "$dir" ]] || { log_error "Not a directory: $dir"; return 1; } [[ -w "$dir" ]] || { log_error "Directory not writable: $dir"; return 1; } } Input Sanitization 1 2 3 4 5 6 7 8 sanitize_filename() { local input=$1 # Remove path components and dangerous characters echo "${input##*/}" | tr -cd '[:alnum:]._-' } # Usage safe_name=$(sanitize_filename "$user_input") Configuration Config File Loading 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 load_config() { local config_file=$1 if [[ -f "$config_file" ]]; then log_debug "Loading config: $config_file" # shellcheck source=/dev/null source "$config_file" else log_warn "Config file not found, using defaults: $config_file" fi } # Config file format (config.sh): # DB_HOST="localhost" # DB_PORT=5432 # ENABLE_FEATURE=true Environment Variable Defaults 1 2 3 4 5 6 : "${DB_HOST:=localhost}" : "${DB_PORT:=5432}" : "${LOG_LEVEL:=info}" # Or with validation DB_HOST="${DB_HOST:?DB_HOST environment variable required}" Parallel Execution Using xargs 1 2 3 4 process_files() { find . -name "*.txt" -print0 | \ xargs -0 -P 4 -I {} bash -c 'process_single_file "$1"' _ {} } Using Background Jobs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 parallel_tasks() { local pids=() for item in "${items[@]}"; do process_item "$item" & pids+=($!) done # Wait for all and check results local failed=0 for pid in "${pids[@]}"; do if ! wait "$pid"; then ((failed++)) fi done [[ $failed -eq 0 ]] || log_error "$failed tasks failed" return $failed } Locking Prevent Concurrent Runs 1 2 3 4 5 6 7 8 9 10 11 LOCK_FILE="/var/run/${SCRIPT_NAME}.lock" acquire_lock() { if ! mkdir "$LOCK_FILE" 2>/dev/null; then log_error "Another instance is running (lock: $LOCK_FILE)" exit 1 fi trap 'rm -rf "$LOCK_FILE"' EXIT } acquire_lock With flock 1 2 3 4 5 6 7 8 9 10 11 12 LOCK_FD=200 LOCK_FILE="/var/run/${SCRIPT_NAME}.lock" acquire_lock() { eval "exec $LOCK_FD>$LOCK_FILE" if ! flock -n $LOCK_FD; then log_error "Another instance is running" exit 1 fi } acquire_lock Testing Assertions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 assert_equals() { local expected=$1 actual=$2 message=${3:-"Assertion failed"} if [[ "$expected" != "$actual" ]]; then log_error "$message: expected '$expected', got '$actual'" return 1 fi } assert_file_exists() { local file=$1 if [[ ! -f "$file" ]]; then log_error "File should exist: $file" return 1 fi } # Usage assert_equals "200" "$status_code" "HTTP status check" assert_file_exists "/tmp/output.txt" Test Mode 1 2 3 4 5 6 7 if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then # Script is being run directly main "$@" else # Script is being sourced (for testing) log_debug "Script sourced, not executing main" fi Debugging Debug Mode 1 [[ "${DEBUG:-}" == true ]] && set -x Trace Function Calls 1 2 3 4 5 6 7 8 trace() { log_debug "TRACE: ${FUNCNAME[1]}(${*})" } my_function() { trace "$@" # function body } Common Pitfalls Quote Everything 1 2 3 4 5 # Bad if [ $var == "value" ]; then # Good if [[ "$var" == "value" ]]; then Use Arrays for Lists 1 2 3 4 5 6 7 # Bad files="file1.txt file2.txt file with spaces.txt" for f in $files; do echo "$f"; done # Good files=("file1.txt" "file2.txt" "file with spaces.txt") for f in "${files[@]}"; do echo "$f"; done Check Command Success 1 2 3 4 5 6 7 8 9 10 11 # Bad output=$(some_command) echo "$output" # Good if output=$(some_command 2>&1); then echo "$output" else log_error "Command failed: $output" exit 1 fi Good bash scripts are defensive, verbose when needed, and fail fast. Start with the template, add what you need, and resist the urge to be clever. ...

Docker Compose bridges the gap between single-container development and full orchestration. These patterns will help you build maintainable, production-ready configurations. Project Structure m ├ ├ ├ ├ ├ ├ └ y ─ ─ ─ ─ ─ ─ ─ p ─ ─ ─ ─ ─ ─ ─ r o d d d d . . s ├ │ ├ │ └ j o o o o e e e ─ ─ ─ e c c c c n n r ─ ─ ─ c k k k k v v v t e e e e . i a └ w └ n ├ └ / r r r r e c p ─ o ─ g ─ ─ - - - - x e p ─ r ─ i ─ ─ c c c c a s k n o o o o m D e D x D n m m m m p o r / o g p p p p l c c c i o o o o e k k k n s s s s e e e x e e e e r r r . . . . . f f f c y o p t i i i o m v r e l l l n l e o s e e e f r d t r . . i y y d m m e l l . y m l # # # # # # B D P T E T a e r e n e s v o s v m e d t i p u r l c v c c o a o e t o n t n r i n m e f r o f e i i n i n ( g d g t c u e u o r s v r v m a e a a m t ( r t r i i a r i i t o u i o a t n t d n b e o e l d - s e ) l s o a d e d ) Base Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 # docker-compose.yml version: "3.8" services: app: build: context: ./services/app dockerfile: Dockerfile environment: - DATABASE_URL=${DATABASE_URL} - REDIS_URL=${REDIS_URL} depends_on: db: condition: service_healthy redis: condition: service_started networks: - backend restart: unless-stopped db: image: postgres:15-alpine environment: POSTGRES_DB: ${DB_NAME} POSTGRES_USER: ${DB_USER} POSTGRES_PASSWORD: ${DB_PASSWORD} volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"] interval: 10s timeout: 5s retries: 5 networks: - backend redis: image: redis:7-alpine volumes: - redis_data:/data networks: - backend networks: backend: driver: bridge volumes: postgres_data: redis_data: Development Overrides 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 # docker-compose.override.yml (auto-loaded with docker-compose up) version: "3.8" services: app: build: target: development volumes: - ./src:/app/src:cached - /app/node_modules ports: - "3000:3000" - "9229:9229" # Debugger environment: - DEBUG=true - LOG_LEVEL=debug command: npm run dev db: ports: - "5432:5432" redis: ports: - "6379:6379" Production Configuration 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 # docker-compose.prod.yml version: "3.8" services: app: build: target: production deploy: replicas: 3 resources: limits: cpus: '1' memory: 512M reservations: cpus: '0.5' memory: 256M restart_policy: condition: on-failure delay: 5s max_attempts: 3 environment: - NODE_ENV=production - LOG_LEVEL=info logging: driver: json-file options: max-size: "10m" max-file: "3" nginx: image: nginx:alpine ports: - "80:80" - "443:443" volumes: - ./services/nginx/nginx.conf:/etc/nginx/nginx.conf:ro - ./certs:/etc/nginx/certs:ro depends_on: - app networks: - backend - frontend networks: frontend: driver: bridge Run with: ...

You know git add, commit, push, and pull. That gets you through 90% of daily work. But the remaining 10%—untangling merge conflicts, finding bug introductions, managing multiple branches simultaneously—requires deeper knowledge. Interactive Rebase The most powerful tool for cleaning up history before sharing. 1 2 3 4 5 # Rebase last 5 commits git rebase -i HEAD~5 # Rebase onto main git rebase -i main The editor opens with commits listed oldest-first: ...

curl is the universal language of HTTP. Every API doc includes curl examples. Every debugging session starts with “can you curl it?” If you’re not comfortable with curl, you’re missing the most portable tool in your kit. Basic Requests 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # GET (default) curl https://api.example.com/users # With headers shown curl -i https://api.example.com/users # Headers only curl -I https://api.example.com/users # Silent (no progress bar) curl -s https://api.example.com/users # Follow redirects curl -L https://example.com/redirect # Verbose (debug mode) curl -v https://api.example.com/users HTTP Methods 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # POST curl -X POST https://api.example.com/users # PUT curl -X PUT https://api.example.com/users/1 # PATCH curl -X PATCH https://api.example.com/users/1 # DELETE curl -X DELETE https://api.example.com/users/1 # HEAD (headers only, like -I) curl -X HEAD https://api.example.com/users Sending Data Form Data 1 2 3 4 5 6 7 # URL-encoded form curl -X POST https://api.example.com/login \ -d "username=admin&password=secret" # From file curl -X POST https://api.example.com/login \ -d @credentials.txt JSON Data 1 2 3 4 5 6 7 8 9 10 11 12 # Inline JSON curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d '{"name":"Alice","email":"alice@example.com"}' # From file curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d @user.json # Using --json (curl 7.82+) curl --json '{"name":"Alice"}' https://api.example.com/users File Upload 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Single file curl -X POST https://api.example.com/upload \ -F "file=@document.pdf" # Multiple files curl -X POST https://api.example.com/upload \ -F "file1=@doc1.pdf" \ -F "file2=@doc2.pdf" # File with custom filename curl -X POST https://api.example.com/upload \ -F "file=@localname.pdf;filename=remote.pdf" # File with content type curl -X POST https://api.example.com/upload \ -F "file=@image.png;type=image/png" # Mixed form data and files curl -X POST https://api.example.com/upload \ -F "title=My Document" \ -F "file=@document.pdf" Headers 1 2 3 4 5 6 7 8 9 10 11 12 13 # Custom header curl -H "X-Custom-Header: value" https://api.example.com # Multiple headers curl -H "Accept: application/json" \ -H "X-API-Version: 2" \ https://api.example.com # User agent curl -A "MyApp/1.0" https://api.example.com # Referer curl -e "https://example.com" https://api.example.com Authentication Basic Auth 1 2 3 4 5 6 7 8 # Username and password curl -u username:password https://api.example.com # Prompt for password curl -u username https://api.example.com # In URL (not recommended) curl https://username:password@api.example.com Bearer Token 1 curl -H "Authorization: Bearer YOUR_TOKEN" https://api.example.com API Key 1 2 3 4 5 # In header curl -H "X-API-Key: YOUR_KEY" https://api.example.com # In query string curl "https://api.example.com?api_key=YOUR_KEY" OAuth 2.0 Flow 1 2 3 4 5 6 7 8 9 # Get access token curl -X POST https://auth.example.com/oauth/token \ -d "grant_type=client_credentials" \ -d "client_id=YOUR_CLIENT_ID" \ -d "client_secret=YOUR_SECRET" # Use token TOKEN="eyJ..." curl -H "Authorization: Bearer $TOKEN" https://api.example.com/resource Digest Auth 1 curl --digest -u username:password https://api.example.com Output Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Save to file curl -o output.html https://example.com # Save with remote filename curl -O https://example.com/file.zip # Save multiple files curl -O https://example.com/file1.zip -O https://example.com/file2.zip # Append to file curl https://example.com >> output.txt # Write headers to file curl -D headers.txt https://example.com # Output to stdout and file curl https://example.com | tee output.html Timeouts and Retries 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Connection timeout (seconds) curl --connect-timeout 5 https://api.example.com # Max time for entire operation curl -m 30 https://api.example.com # Retry on failure curl --retry 3 https://api.example.com # Retry with delay curl --retry 3 --retry-delay 5 https://api.example.com # Retry on specific errors curl --retry 3 --retry-all-errors https://api.example.com SSL/TLS 1 2 3 4 5 6 7 8 9 10 11 12 # Skip certificate verification (insecure!) curl -k https://self-signed.example.com # Use specific CA certificate curl --cacert /path/to/ca.crt https://api.example.com # Client certificate curl --cert client.crt --key client.key https://api.example.com # Force TLS version curl --tlsv1.2 https://api.example.com curl --tlsv1.3 https://api.example.com Proxy 1 2 3 4 5 6 7 8 9 10 11 # HTTP proxy curl -x http://proxy:8080 https://api.example.com # SOCKS5 proxy curl --socks5 localhost:1080 https://api.example.com # Proxy with auth curl -x http://user:pass@proxy:8080 https://api.example.com # No proxy for specific hosts curl --noproxy "localhost,*.internal" https://api.example.com Cookies 1 2 3 4 5 6 7 8 9 10 11 # Send cookie curl -b "session=abc123" https://api.example.com # Send cookies from file curl -b cookies.txt https://api.example.com # Save cookies to file curl -c cookies.txt https://api.example.com/login # Full session (save and send) curl -b cookies.txt -c cookies.txt https://api.example.com Response Inspection 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # HTTP status code only curl -s -o /dev/null -w "%{http_code}" https://api.example.com # Response time curl -s -o /dev/null -w "%{time_total}s" https://api.example.com # Detailed timing curl -s -o /dev/null -w " DNS: %{time_namelookup}s Connect: %{time_connect}s TLS: %{time_appconnect}s Start: %{time_starttransfer}s Total: %{time_total}s Size: %{size_download} bytes Speed: %{speed_download} bytes/sec " https://api.example.com # Content type curl -s -o /dev/null -w "%{content_type}" https://api.example.com Scripting Patterns Health Check 1 2 3 4 5 6 7 8 9 10 11 #!/bin/bash URL="https://api.example.com/health" STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$URL") if [ "$STATUS" -eq 200 ]; then echo "OK" exit 0 else echo "FAIL: HTTP $STATUS" exit 1 fi API Wrapper 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #!/bin/bash API_BASE="https://api.example.com" API_KEY="${API_KEY:?API_KEY required}" api_get() { curl -s -H "Authorization: Bearer $API_KEY" "$API_BASE$1" } api_post() { curl -s -X POST \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$2" \ "$API_BASE$1" } # Usage api_get "/users" | jq '.' api_post "/users" '{"name":"Alice"}' Retry with Backoff 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 #!/bin/bash URL="$1" MAX_RETRIES=5 RETRY_DELAY=1 for i in $(seq 1 $MAX_RETRIES); do RESPONSE=$(curl -s -w "\n%{http_code}" "$URL") STATUS=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') if [ "$STATUS" -eq 200 ]; then echo "$BODY" exit 0 fi echo "Attempt $i failed (HTTP $STATUS), retrying in ${RETRY_DELAY}s..." >&2 sleep $RETRY_DELAY RETRY_DELAY=$((RETRY_DELAY * 2)) done echo "Failed after $MAX_RETRIES attempts" >&2 exit 1 Parallel Requests 1 2 3 4 5 # Using xargs cat urls.txt | xargs -P 10 -I {} curl -s -o /dev/null -w "{}: %{http_code}\n" {} # Using GNU parallel parallel -j 10 curl -s -o /dev/null -w "{}: %{http_code}\n" ::: $(cat urls.txt) Config Files Create ~/.curlrc for defaults: ...