Linux

When a server is slow and people are yelling, you need a systematic approach. Here’s what to run in the first five minutes. The Checklist 1 2 3 4 5 6 7 8 uptime dmesg | tail vmstat 1 5 mpstat -P ALL 1 3 pidstat 1 3 iostat -xz 1 3 free -h sar -n DEV 1 3 Let’s break down what each tells you. 1. uptime 1 2 $ uptime 16:30:01 up 45 days, 3:22, 2 users, load average: 8.42, 6.31, 5.12 Load averages: 1-minute, 5-minute, 15-minute. ...

Bash scripts glue systems together. Here’s how to write them without the usual fragility. Script Header Always start with: 1 2 3 4 5 6 #!/usr/bin/env bash set -euo pipefail # -e: Exit on error # -u: Error on undefined variables # -o pipefail: Fail if any pipe command fails Argument Parsing Simple Positional 1 2 3 4 5 6 7 8 9 #!/usr/bin/env bash set -euo pipefail if [[ $# -lt 1 ]]; then echo "Usage: $0 <filename>" >&2 exit 1 fi FILENAME="$1" With Options (getopts) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 #!/usr/bin/env bash set -euo pipefail usage() { echo "Usage: $0 [-v] [-o output] [-n count] input" echo " -v Verbose mode" echo " -o output Output file" echo " -n count Number of iterations" exit 1 } VERBOSE=false OUTPUT="" COUNT=1 while getopts "vo:n:h" opt; do case $opt in v) VERBOSE=true ;; o) OUTPUT="$OPTARG" ;; n) COUNT="$OPTARG" ;; h) usage ;; *) usage ;; esac done shift $((OPTIND - 1)) if [[ $# -lt 1 ]]; then usage fi INPUT="$1" Long Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #!/usr/bin/env bash set -euo pipefail VERBOSE=false OUTPUT="" while [[ $# -gt 0 ]]; do case $1 in -v|--verbose) VERBOSE=true shift ;; -o|--output) OUTPUT="$2" shift 2 ;; -h|--help) usage ;; -*) echo "Unknown option: $1" >&2 exit 1 ;; *) break ;; esac done Error Handling Trap for Cleanup 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #!/usr/bin/env bash set -euo pipefail TMPDIR="" cleanup() { if [[ -n "$TMPDIR" && -d "$TMPDIR" ]]; then rm -rf "$TMPDIR" fi } trap cleanup EXIT TMPDIR=$(mktemp -d) # Work with $TMPDIR - it's cleaned up on exit, error, or interrupt Custom Error Handler 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #!/usr/bin/env bash set -euo pipefail error() { echo "Error: $1" >&2 exit "${2:-1}" } warn() { echo "Warning: $1" >&2 } # Usage [[ -f "$CONFIG" ]] || error "Config file not found: $CONFIG" Detailed Error Reporting 1 2 3 4 5 6 7 8 #!/usr/bin/env bash set -euo pipefail on_error() { echo "Error on line $1" >&2 exit 1 } trap 'on_error $LINENO' ERR Variables and Defaults 1 2 3 4 5 6 7 8 9 10 11 # Default value NAME="${1:-default}" # Error if unset NAME="${1:?Error: name required}" # Default only if unset (not empty) NAME="${NAME-default}" # Assign default if unset : "${NAME:=default}" String Operations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 FILE="/path/to/file.txt" # Extract parts echo "${FILE##*/}" # file.txt (basename) echo "${FILE%/*}" # /path/to (dirname) echo "${FILE%.txt}" # /path/to/file (remove extension) echo "${FILE##*.}" # txt (extension only) # Replace echo "${FILE/path/new}" # /new/to/file.txt echo "${FILE//t/T}" # /paTh/To/file.TxT (all occurrences) # Case conversion echo "${FILE^^}" # Uppercase echo "${FILE,,}" # Lowercase # Length echo "${#FILE}" # String length Conditionals 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 # File tests [[ -f "$FILE" ]] # File exists [[ -d "$DIR" ]] # Directory exists [[ -r "$FILE" ]] # Readable [[ -w "$FILE" ]] # Writable [[ -x "$FILE" ]] # Executable [[ -s "$FILE" ]] # Non-empty # String tests [[ -z "$VAR" ]] # Empty [[ -n "$VAR" ]] # Non-empty [[ "$A" == "$B" ]] # Equal [[ "$A" != "$B" ]] # Not equal [[ "$A" =~ regex ]] # Regex match # Numeric tests [[ $A -eq $B ]] # Equal [[ $A -ne $B ]] # Not equal [[ $A -lt $B ]] # Less than [[ $A -le $B ]] # Less or equal [[ $A -gt $B ]] # Greater than [[ $A -ge $B ]] # Greater or equal # Logical [[ $A && $B ]] # And [[ $A || $B ]] # Or [[ ! $A ]] # Not Loops 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 # Over arguments for arg in "$@"; do echo "$arg" done # Over array arr=("one" "two" "three") for item in "${arr[@]}"; do echo "$item" done # C-style for ((i=0; i<10; i++)); do echo "$i" done # Over files for file in *.txt; do [[ -f "$file" ]] || continue echo "$file" done # Read lines from file while IFS= read -r line; do echo "$line" done < "$FILE" # Read lines from command while IFS= read -r line; do echo "$line" done < <(some_command) Functions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 # Basic function greet() { local name="$1" echo "Hello, $name" } # With return value is_valid() { local input="$1" [[ "$input" =~ ^[0-9]+$ ]] } if is_valid "$value"; then echo "Valid" fi # Return data via stdout get_config() { cat /etc/myapp/config } CONFIG=$(get_config) # Local variables process() { local tmp tmp=$(mktemp) # tmp is local to this function } Arrays 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Create arr=("one" "two" "three") arr[3]="four" # Access echo "${arr[0]}" # First element echo "${arr[@]}" # All elements echo "${#arr[@]}" # Length echo "${arr[@]:1:2}" # Slice (start:length) # Add arr+=("five") # Iterate for item in "${arr[@]}"; do echo "$item" done # With index for i in "${!arr[@]}"; do echo "$i: ${arr[$i]}" done Associative Arrays 1 2 3 4 5 6 7 8 9 10 11 12 declare -A config config[host]="localhost" config[port]="8080" echo "${config[host]}" echo "${!config[@]}" # All keys echo "${config[@]}" # All values for key in "${!config[@]}"; do echo "$key: ${config[$key]}" done Process Substitution 1 2 3 4 5 6 7 # Compare two commands diff <(sort file1) <(sort file2) # Feed command output as file while read -r line; do echo "$line" done < <(curl -s "$URL") Subshells 1 2 3 4 5 6 7 8 9 10 11 12 # Run in subshell (changes don't affect parent) ( cd /tmp rm -f *.tmp ) # Still in original directory # Capture output result=$(command) # Capture with error result=$(command 2>&1) Here Documents 1 2 3 4 5 6 7 8 9 10 11 12 13 # Multi-line string cat << EOF This is a multi-line string with $VARIABLE expansion EOF # No variable expansion cat << 'EOF' This preserves $VARIABLE literally EOF # Here string grep "pattern" <<< "$string" Practical Patterns Check Dependencies 1 2 3 4 5 6 7 8 9 check_deps() { local deps=("curl" "jq" "git") for dep in "${deps[@]}"; do if ! command -v "$dep" &> /dev/null; then echo "Missing dependency: $dep" >&2 exit 1 fi done } Logging 1 2 3 4 5 6 7 LOG_FILE="/var/log/myapp.log" log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE" } log "Starting process" Confirmation Prompt 1 2 3 4 5 6 7 8 confirm() { read -rp "$1 [y/N] " response [[ "$response" =~ ^[Yy]$ ]] } if confirm "Delete all files?"; then rm -rf /tmp/data/* fi Retry Logic 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 retry() { local max_attempts="$1" local delay="$2" shift 2 local attempt=1 until "$@"; do if ((attempt >= max_attempts)); then echo "Failed after $attempt attempts" >&2 return 1 fi echo "Attempt $attempt failed, retrying in ${delay}s..." sleep "$delay" ((attempt++)) done } retry 3 5 curl -sf "$URL" Lock File 1 2 3 4 5 6 7 8 9 10 11 12 LOCKFILE="/var/lock/myapp.lock" acquire_lock() { exec 200>"$LOCKFILE" flock -n 200 || { echo "Another instance is running" >&2 exit 1 } } acquire_lock # Script continues only if lock acquired Complete Example 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 #!/usr/bin/env bash set -euo pipefail readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly SCRIPT_NAME="$(basename "$0")" usage() { cat << EOF Usage: $SCRIPT_NAME [options] <input> Options: -o, --output FILE Output file (default: stdout) -v, --verbose Verbose output -h, --help Show this help Examples: $SCRIPT_NAME data.txt $SCRIPT_NAME -o result.txt -v input.txt EOF exit "${1:-0}" } log() { if [[ "$VERBOSE" == true ]]; then echo "[$(date '+%H:%M:%S')] $*" >&2 fi } error() { echo "Error: $*" >&2 exit 1 } # Defaults VERBOSE=false OUTPUT="/dev/stdout" # Parse arguments while [[ $# -gt 0 ]]; do case $1 in -o|--output) OUTPUT="$2"; shift 2 ;; -v|--verbose) VERBOSE=true; shift ;; -h|--help) usage ;; -*) error "Unknown option: $1" ;; *) break ;; esac done [[ $# -ge 1 ]] || usage 1 INPUT="$1" # Validate [[ -f "$INPUT" ]] || error "File not found: $INPUT" # Main logic log "Processing $INPUT" process_data < "$INPUT" > "$OUTPUT" log "Done" Bash scripts don’t have to be fragile. Apply these patterns and they’ll work reliably for years. ...

rsync is the standard for efficient file transfer. It only copies what changed, handles interruptions gracefully, and works over SSH. Here’s how to use it well. Basic Syntax 1 rsync [options] source destination The trailing slash matters: 1 2 rsync -av src/ dest/ # Contents of src into dest rsync -av src dest/ # Directory src into dest (creates dest/src/) Essential Options 1 2 3 4 5 6 -a, --archive # Archive mode (preserves permissions, timestamps, etc.) -v, --verbose # Show what's being transferred -z, --compress # Compress during transfer -P # Progress + partial (resume interrupted transfers) --delete # Remove files from dest that aren't in source -n, --dry-run # Show what would happen Common Patterns Local Backup 1 2 3 4 5 # Mirror directory rsync -av --delete /home/user/documents/ /backup/documents/ # Dry run first rsync -avn --delete /home/user/documents/ /backup/documents/ Remote Sync Over SSH 1 2 3 4 5 6 7 8 # Push to remote rsync -avz -e ssh /local/dir/ user@server:/remote/dir/ # Pull from remote rsync -avz -e ssh user@server:/remote/dir/ /local/dir/ # Custom SSH port rsync -avz -e "ssh -p 2222" /local/ user@server:/remote/ With Progress 1 2 3 4 5 # Single file progress rsync -avP largefile.zip server:/dest/ # Overall progress (rsync 3.1+) rsync -av --info=progress2 /source/ /dest/ Exclusions 1 2 3 4 5 6 7 8 # Exclude patterns rsync -av --exclude='*.log' --exclude='tmp/' /source/ /dest/ # Exclude from file rsync -av --exclude-from='exclude.txt' /source/ /dest/ # Include only certain files rsync -av --include='*.py' --exclude='*' /source/ /dest/ Example exclude file: ...

Cron works. It’s also from 1975. systemd timers offer logging integration, dependency handling, and more flexible scheduling. Here’s how to use them. Why Timers Over Cron? Logging: Output goes to journald automatically Dependencies: Wait for network, mounts, or other services Flexibility: Calendar events, monotonic timers, randomized delays Visibility: systemctl list-timers shows everything Consistency: Same management as other systemd units Basic Structure A timer needs two files: A .timer unit (the schedule) A .service unit (the job) Place them in /etc/systemd/system/ (system-wide) or ~/.config/systemd/user/ (user). ...

awk sits between grep and a full programming language. It’s perfect for columnar data, log files, and quick text transformations. The Basic Pattern 1 awk 'pattern { action }' file If pattern matches, run action. No pattern means every line. No action means print. 1 2 3 4 5 6 7 8 9 10 11 # Print everything awk '{ print }' file.txt # Print lines matching pattern awk '/error/' file.txt # Print second column awk '{ print $2 }' file.txt # Combined: errors, show timestamp and message awk '/error/ { print $1, $4 }' app.log Field Handling awk splits lines into fields by whitespace (default): ...

SSH connections drop. Scripts need to run overnight. You need six terminals for one task. tmux solves all of this. Why tmux? Persistence: Sessions survive disconnects Multiplexing: Multiple windows and panes in one connection Sharing: Pair programming on the same session Automation: Scriptable terminal control Getting Started 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Start a new session tmux # Start named session tmux new -s work # Detach (inside tmux) Ctrl+b d # List sessions tmux ls # Reattach tmux attach -t work The prefix key is Ctrl+b by default. Press it, then the command key. ...

JSON is everywhere. APIs return it, configs use it, logs contain it. jq is the Swiss Army knife for processing it all from the command line. Basic Selection 1 2 3 4 5 6 7 8 9 10 11 12 13 # Pretty print echo '{"name":"alice","age":30}' | jq . # Extract a field echo '{"name":"alice","age":30}' | jq '.name' # Output: "alice" # Raw output (no quotes) echo '{"name":"alice","age":30}' | jq -r '.name' # Output: alice # Nested fields echo '{"user":{"name":"alice"}}' | jq '.user.name' Working with Arrays 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 # Get all elements echo '[1,2,3]' | jq '.[]' # Output: # 1 # 2 # 3 # Get specific index echo '["a","b","c"]' | jq '.[1]' # Output: "b" # Slice echo '[1,2,3,4,5]' | jq '.[2:4]' # Output: [3,4] # First/last echo '[1,2,3]' | jq 'first' # 1 echo '[1,2,3]' | jq 'last' # 3 Filtering 1 2 3 4 5 6 7 8 9 10 11 12 13 # Select objects matching condition echo '[{"name":"alice","age":30},{"name":"bob","age":25}]' | \ jq '.[] | select(.age > 27)' # Output: {"name":"alice","age":30} # Multiple conditions jq '.[] | select(.status == "active" and .role == "admin")' # Contains jq '.[] | select(.tags | contains(["important"]))' # Regex matching jq '.[] | select(.email | test("@company\\.com$"))' Transforming Data 1 2 3 4 5 6 7 8 9 10 11 12 # Create new object echo '{"first":"Alice","last":"Smith"}' | \ jq '{fullName: (.first + " " + .last)}' # Output: {"fullName":"Alice Smith"} # Map over array echo '[1,2,3]' | jq 'map(. * 2)' # Output: [2,4,6] # Transform array of objects echo '[{"name":"alice"},{"name":"bob"}]' | \ jq 'map({user: .name, active: true})' API Response Processing 1 2 3 4 5 6 7 8 9 10 11 12 # Extract data from GitHub API curl -s https://api.github.com/users/torvalds/repos | \ jq '.[] | {name, stars: .stargazers_count, language}' | \ jq -s 'sort_by(.stars) | reverse | .[0:5]' # Get just names curl -s https://api.github.com/users/torvalds/repos | \ jq -r '.[].name' # Count items curl -s https://api.github.com/users/torvalds/repos | \ jq 'length' Aggregation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Sum echo '[{"value":10},{"value":20},{"value":30}]' | \ jq '[.[].value] | add' # Output: 60 # Average jq '[.[].value] | add / length' # Group by echo '[{"type":"a","n":1},{"type":"b","n":2},{"type":"a","n":3}]' | \ jq 'group_by(.type) | map({type: .[0].type, total: [.[].n] | add})' # Count by field jq 'group_by(.status) | map({status: .[0].status, count: length})' Building Output 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # Concatenate to string echo '{"host":"db","port":5432}' | \ jq -r '"\(.host):\(.port)"' # Output: db:5432 # Create CSV echo '[{"name":"alice","age":30},{"name":"bob","age":25}]' | \ jq -r '.[] | [.name, .age] | @csv' # Output: # "alice",30 # "bob",25 # Create TSV jq -r '.[] | [.name, .age] | @tsv' # URI encode jq -r '@uri' # Base64 jq -r '@base64' Conditional Logic 1 2 3 4 5 6 7 8 9 10 11 # If/then/else echo '{"score":85}' | \ jq 'if .score >= 90 then "A" elif .score >= 80 then "B" else "C" end' # Alternative operator (default values) echo '{"name":"alice"}' | jq '.age // 0' # Output: 0 # Try (suppress errors) echo '{"a":1}' | jq '.b.c.d // "missing"' # Output: "missing" Modifying JSON 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Update field echo '{"name":"alice","age":30}' | jq '.age = 31' # Add field echo '{"name":"alice"}' | jq '. + {active: true}' # Delete field echo '{"name":"alice","temp":123}' | jq 'del(.temp)' # Update nested echo '{"user":{"name":"alice"}}' | jq '.user.name = "bob"' # Recursive update jq '.. | objects | .timestamp |= (. // now)' Multiple Files 1 2 3 4 5 6 7 8 9 # Combine objects from files jq -s '.[0] * .[1]' defaults.json overrides.json # Process files independently jq -r '.name' file1.json file2.json # Slurp into array jq -s '.' file1.json file2.json # Output: [{...}, {...}] Stream Processing For large files, use streaming: ...

If you’re still typing ssh -i ~/.ssh/mykey.pem ec2-user@ec2-54-123-45-67.compute-1.amazonaws.com, you’re working too hard. SSH config transforms verbose commands into simple ssh prod invocations. The Basics Create or edit ~/.ssh/config: H o s t H U I p o s d r s e e o t r n d N t a e i m c t e 2 y - F 5 u i 4 s l . e e 1 r 2 ~ 3 / . . 4 s 5 s . h 6 / 7 p r o d - k e y . p e m Now ssh prod connects with the right key and user. No more remembering IP addresses. ...

top works. htop works better. It’s colorful, interactive, and actually pleasant to use. Here’s how to get the most from it. Installation 1 2 3 4 5 6 7 8 # Debian/Ubuntu sudo apt install htop # RHEL/CentOS/Fedora sudo dnf install htop # macOS brew install htop The Interface Launch with htop. You’ll see: Top section: CPU bars (one per core) Memory and swap usage Tasks, load average, uptime Process list: ...

You know grep "error" logfile.txt. But grep can do so much more — recursive searches, context lines, inverse matching, and regex patterns that turn hours of manual searching into seconds. The Basics 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Search for pattern in file grep "error" app.log # Case-insensitive grep -i "error" app.log # Show line numbers grep -n "error" app.log # Count matches grep -c "error" app.log # Only show filenames with matches grep -l "error" *.log Recursive Search 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Search all files in directory tree grep -r "TODO" ./src # With line numbers grep -rn "TODO" ./src # Include only certain files grep -r --include="*.py" "import os" . # Exclude directories grep -r --exclude-dir=node_modules "console.log" . # Multiple excludes grep -r --exclude-dir={node_modules,.git,dist} "function" . Context Lines When you find a match, you often need surrounding context: ...