Posts

When GitHub-hosted runners aren’t enough—when you need GPU access, specific hardware, private network connectivity, or just want to stop paying per-minute—self-hosted runners are the answer. Why Self-Hosted? Performance: Your hardware, your speed. No cold starts, local caching, faster artifact access. Cost: After a certain threshold, self-hosted is dramatically cheaper. GitHub-hosted minutes add up fast for active repos. Access: Private networks, internal services, specialized hardware, air-gapped environments. Control: Exact OS versions, pre-installed dependencies, custom security configurations. ...

systemd’s journal collects logs from all services, the kernel, and system messages in one place. journalctl is your tool for searching, filtering, and following those logs. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # Show all logs (oldest first) journalctl # Show all logs (newest first) journalctl -r # Follow new entries (like tail -f) journalctl -f # Show only errors and above journalctl -p err Filter by Time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # Since boot journalctl -b # Previous boot journalctl -b -1 # Since specific time journalctl --since "2024-02-25 10:00:00" # Until specific time journalctl --until "2024-02-25 12:00:00" # Time range journalctl --since "1 hour ago" journalctl --since "2024-02-25" --until "2024-02-26" # Relative times journalctl --since "yesterday" journalctl --since "10 minutes ago" Filter by Unit (Service) 1 2 3 4 5 6 7 8 9 10 11 # Specific service journalctl -u nginx # Multiple services journalctl -u nginx -u php-fpm # Follow specific service journalctl -u nginx -f # Service since boot journalctl -u nginx -b Filter by Priority Priority levels (0=emergency to 7=debug): ...

strace intercepts and records system calls made by a process. When a program hangs, crashes, or behaves mysteriously, strace reveals what it’s actually doing at the kernel level. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # Trace a command strace ls # Trace running process strace -p 1234 # Trace with timestamps strace -t ls # Trace with relative timestamps strace -r ls Filtering System Calls 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Only file operations strace -e trace=file ls # Only network operations strace -e trace=network curl example.com # Only process operations strace -e trace=process bash -c 'sleep 1' # Specific syscalls strace -e open,read,write cat file.txt # Exclude syscalls strace -e trace=!mmap ls Trace Categories 1 2 3 4 5 6 7 file # open, stat, chmod, etc. process # fork, exec, exit, etc. network # socket, connect, send, etc. signal # signal, kill, etc. ipc # shmget, semop, etc. desc # read, write, close, etc. memory # mmap, brk, etc. Output Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Write to file strace -o output.txt ls # Append to file strace -o output.txt -A ls # With timestamps (wall clock) strace -t ls # With microseconds strace -tt ls # Relative timestamps strace -r ls Following Children 1 2 3 4 5 6 # Follow forked processes strace -f bash -c 'ls; echo done' # Follow forks and write separate files strace -ff -o trace ls # Creates trace.1234, trace.1235, etc. String Output 1 2 3 4 5 # Show full strings (default truncates at 32 chars) strace -s 1000 cat file.txt # Show full strings for specific calls strace -e read -s 10000 cat file.txt Statistics 1 2 3 4 5 6 7 8 9 # Summary of syscalls strace -c ls # Sample output: # % time seconds usecs/call calls errors syscall # ------ ----------- ----------- --------- --------- ---------------- # 45.00 0.000045 45 1 execve # 25.00 0.000025 3 8 mmap # 15.00 0.000015 2 6 openat 1 2 3 # Summary with detailed trace strace -c -S time ls # Sort by time strace -c -S calls ls # Sort by call count Practical Examples Debug “File Not Found” 1 2 # See what files the program is trying to open strace -e openat ./myprogram 2>&1 | grep -i "no such file" Find Config File Locations 1 2 # See all files a program tries to read strace -e openat nginx -t 2>&1 | grep -E "openat.*O_RDONLY" Debug Connection Issues 1 2 3 4 5 # Watch network connections strace -e connect curl https://example.com # See DNS lookups strace -e socket,connect,sendto,recvfrom dig example.com Debug Hanging Process 1 2 3 4 5 6 7 # Attach to hung process strace -p $(pgrep hung-process) # Common findings: # - Waiting on read() = blocked on input # - Waiting on futex() = waiting for lock # - Waiting on poll/select = waiting for I/O Find Why Program is Slow 1 2 3 4 5 6 7 8 # Time each syscall strace -T ls # Shows time spent in each call: # openat(AT_FDCWD, ".", ...) = 3 <0.000015> # Summary to find slow operations strace -c -S time slow-program Debug Permission Issues 1 2 3 4 5 # See access denials strace -e openat,access ./program 2>&1 | grep -i denied # Sample output: # openat(AT_FDCWD, "/etc/secret", O_RDONLY) = -1 EACCES (Permission denied) Watch File I/O 1 2 3 4 5 # See all reads and writes strace -e read,write -s 100 cat file.txt # Count I/O operations strace -c -e read,write dd if=/dev/zero of=/dev/null bs=1M count=100 Debug Signal Handling 1 2 3 4 5 # Trace signals strace -e signal,kill ./program # See what signal killed a process strace -e trace=signal -p 1234 Find Library Loading Issues 1 2 3 4 5 6 # See shared library loading strace -e openat ./program 2>&1 | grep "\.so" # Common issues: # - Library not found # - Wrong library version loaded Advanced Usage Inject Faults 1 2 3 4 5 # Make open() fail with ENOENT strace -e fault=openat:error=ENOENT ls # Fail every 3rd call strace -e fault=read:error=EIO:when=3 cat file.txt Decode Arguments 1 2 3 4 5 # Decode socket addresses strace -yy curl example.com # Decode file descriptors strace -y cat file.txt Trace Specific Syscall Return 1 2 3 4 5 # Only show failed syscalls strace -Z ls /nonexistent # Show syscalls that return specific value strace -e status=failed ls /nonexistent Reading strace Output o │ │ │ │ │ └ p ─ e ─ n a S t y ( s A t T │ │ │ │ └ e _ ─ m F ─ D c C D a W i l D r l , e c n " t a / o m e r e t y c │ │ │ └ / ─ f p ─ d a s P ( s a A w t T d h _ " n F , a D m C O e W _ D R │ │ └ a D ─ r = O ─ g N u c L F m u Y l e r ) a n r g t e = s n t 3 │ └ d ─ i ─ r ) R e t u r n v a l u e ( f d 3 ) Common Return Values = = = = = = = 0 N - - - - - 1 1 1 1 1 E E E E E N A A E I O C G X N E C A I T N E I S R T S N T S F N P R F I u i o e e i n c l r s l t c e s m o e e e u i u r s d c s r e r s e h s c x u s i e i p ( c f o s t f r i n t t e o i l e s d r p e d m t e p s m o o n o y a r r i r s n e a t y o d d r e r i i m c r l a c e y c l o c a l u t u l s n o n l ) t r a y v a i l a b l e Alternatives 1 2 3 4 5 6 7 8 # ltrace - trace library calls (not syscalls) ltrace ./program # perf trace - lower overhead perf trace ls # bpftrace - more powerful, requires setup bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s\n", str(args->filename)); }' Performance Note strace adds significant overhead — programs run much slower when traced. For production debugging: ...

lsof (list open files) shows which processes have which files open. Since Unix treats everything as a file — including network connections, devices, and pipes — lsof is incredibly powerful for system debugging. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # List all open files (overwhelming) lsof # List open files for specific file lsof /var/log/syslog # List open files for user lsof -u username # List open files for process lsof -p 1234 Find What’s Using a File 1 2 3 4 5 6 7 8 # Who has this file open? lsof /path/to/file # Who has files open in this directory? lsof +D /var/log # Recursive directory search lsof +D /var/log/ “Device or resource busy” Debugging 1 2 3 4 5 # Can't unmount? Find what's using it lsof +D /mnt/usb # Find open files on filesystem lsof /dev/sda1 Find What’s Using a Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # What's on port 8080? lsof -i :8080 # What's on port 80 or 443? lsof -i :80 -i :443 # All network connections lsof -i # TCP only lsof -i TCP # UDP only lsof -i UDP Specific Protocol and Port 1 2 3 4 5 6 7 8 # TCP port 22 lsof -i TCP:22 # UDP port 53 lsof -i UDP:53 # Port range lsof -i :1-1024 Filter by Process 1 2 3 4 5 6 7 8 9 10 11 # By PID lsof -p 1234 # By process name lsof -c nginx # By multiple process names lsof -c nginx -c apache # Exclude process lsof -c ^nginx Filter by User 1 2 3 4 5 6 7 8 9 10 11 # By username lsof -u www-data # By UID lsof -u 1000 # Multiple users lsof -u user1,user2 # Exclude user lsof -u ^root Network Connections List All Connections 1 2 3 4 5 6 7 8 # All internet connections lsof -i # Established connections only lsof -i | grep ESTABLISHED # Listening sockets lsof -i | grep LISTEN Connections to Specific Host 1 2 3 4 5 6 7 8 # Connections to specific IP lsof -i @192.168.1.100 # Connections to hostname lsof -i @example.com # Connections to specific host and port lsof -i @192.168.1.100:22 IPv4 vs IPv6 1 2 3 4 5 # IPv4 only lsof -i 4 # IPv6 only lsof -i 6 Output Formatting 1 2 3 4 5 6 7 8 9 10 # Terse output (parseable) lsof -t /path/to/file # Returns just PIDs # No header lsof +D /var/log | tail -n +2 # Specific fields lsof -F p /path/to/file # PID only lsof -F c /path/to/file # Command only Combining Filters By default, filters are OR’d. Use -a to AND them: ...

Unix permissions determine who can do what with files and directories. Understanding them is fundamental to system security and avoiding “Permission denied” errors. The Permission Model Every file has three permission types for three user classes: Permission Types: r (read) = 4 w (write) = 2 x (execute) = 1 User Classes: u (user/owner) g (group) o (others) Reading Permissions 1 2 ls -l file.txt # -rw-r--r-- 1 user group 1234 Feb 25 12:00 file.txt Breaking down -rw-r--r--: ...

tar (tape archive) bundles files and directories into a single file. Combined with compression, it’s the standard way to package and distribute files on Unix systems. The Basics 1 2 3 4 5 6 7 8 # Create archive tar -cvf archive.tar /path/to/files # Extract archive tar -xvf archive.tar # List contents tar -tvf archive.tar Understanding the Flags 1 2 3 4 5 c = Create archive x = Extract archive t = List contents v = Verbose (show files) f = File (next arg is filename) So tar -cvf = Create, Verbose, File. ...

rsync synchronizes files between locations — local to local, local to remote, remote to local. It’s smart: it only transfers what’s changed, making it fast for incremental backups and deployments. Basic Syntax 1 rsync [options] source destination Local Sync 1 2 3 4 5 6 7 8 9 # Copy directory rsync -av /source/dir/ /dest/dir/ # Copy directory (trailing slash matters!) rsync -av /source/dir /dest/ # Creates /dest/dir/ rsync -av /source/dir/ /dest/ # Contents into /dest/ # Dry run (show what would happen) rsync -avn /source/ /dest/ Remote Sync (SSH) 1 2 3 4 5 6 7 8 9 10 11 # Local to remote rsync -av /local/dir/ user@remote:/remote/dir/ # Remote to local rsync -av user@remote:/remote/dir/ /local/dir/ # Different SSH port rsync -av -e 'ssh -p 2222' /local/ user@remote:/remote/ # With SSH key rsync -av -e 'ssh -i ~/.ssh/mykey' /local/ user@remote:/remote/ Common Options 1 2 3 4 5 6 7 8 9 -a, --archive Archive mode (preserves permissions, timestamps, etc.) -v, --verbose Verbose output -n, --dry-run Show what would be transferred -z, --compress Compress during transfer -P Progress + partial (resume interrupted transfers) --progress Show progress --delete Delete files in dest not in source -r, --recursive Recurse into directories -h, --human-readable Human-readable sizes Archive Mode (-a) -a is equivalent to -rlptgoD: ...

netcat (nc) does one thing: move bytes over a network. That simplicity makes it incredibly versatile — port scanning, file transfers, chat servers, proxies, and network debugging all become one-liners. Basic Usage 1 2 3 4 5 6 7 8 # Connect to host:port nc hostname 80 # Listen on port nc -l 8080 # Listen (keep listening after disconnect) nc -lk 8080 Test if Port is Open 1 2 3 4 5 6 7 8 9 # Quick port check nc -zv hostname 22 # Connection to hostname 22 port [tcp/ssh] succeeded! # Scan port range nc -zv hostname 20-25 # With timeout nc -zv -w 3 hostname 443 Simple Client-Server Communication Server (listener) 1 nc -l 9000 Client 1 nc localhost 9000 Now type in either terminal — text flows both ways. Ctrl+C to exit. ...

watch runs a command every N seconds and displays the output. It’s the simplest form of real-time monitoring — no setup, no configuration, just instant feedback loops. Basic Usage 1 2 3 4 5 6 7 8 # Run command every 2 seconds (default) watch date # Run every 5 seconds watch -n 5 date # Run every 0.5 seconds watch -n 0.5 date Highlight Changes 1 2 3 4 5 # Highlight differences between updates watch -d df -h # Highlight changes permanently (cumulative) watch -d=cumulative df -h Common Options 1 2 3 4 5 6 7 -n, --interval Seconds between updates (default: 2) -d, --differences Highlight changes -t, --no-title Hide the header -b, --beep Beep on command error -e, --errexit Exit on command error -c, --color Interpret ANSI color sequences -x, --exec Pass command to exec instead of sh -c Practical Examples Disk Space 1 2 3 4 5 6 7 8 # Watch disk usage watch df -h # Watch specific mount watch 'df -h | grep /dev/sda1' # Watch directory size watch 'du -sh /var/log' Memory 1 2 3 4 5 # Memory stats watch free -h # Memory with buffers/cache detail watch 'free -h && echo && cat /proc/meminfo | head -10' Processes 1 2 3 4 5 6 7 8 # Process count watch 'ps aux | wc -l' # Specific process watch 'ps aux | grep nginx' # Process memory watch 'ps aux --sort=-%mem | head -10' Network 1 2 3 4 5 6 7 8 9 10 11 # Network connections watch 'netstat -tuln' # Connection count watch 'netstat -an | wc -l' # Active connections watch 'ss -s' # Interface stats watch 'cat /proc/net/dev' Files and Directories 1 2 3 4 5 6 7 8 9 10 11 # File list watch ls -la # Directory size changes watch 'du -sh *' # File count watch 'find . -type f | wc -l' # Recent files watch 'ls -lt | head -10' Docker 1 2 3 4 5 6 7 8 # Container status watch docker ps # Container stats watch 'docker stats --no-stream' # Image list watch docker images Kubernetes 1 2 3 4 5 6 7 8 # Pod status watch kubectl get pods # All resources watch 'kubectl get pods,svc,deploy' # Pod logs (last 5 lines) watch 'kubectl logs -l app=myapp --tail=5' Git 1 2 3 4 5 6 7 8 # Branch status watch git status # Log (one line per commit) watch 'git log --oneline -10' # Diff stats watch git diff --stat Logs 1 2 3 4 5 6 7 8 # Last log lines watch 'tail -5 /var/log/syslog' # Error count watch 'grep -c ERROR /var/log/app.log' # Recent errors watch 'grep ERROR /var/log/app.log | tail -5' APIs and Services 1 2 3 4 5 6 7 8 # HTTP health check watch 'curl -s localhost:8080/health' # API response time watch 'curl -s -w "%{time_total}\n" -o /dev/null http://localhost/api' # Service status watch systemctl status nginx Database 1 2 3 4 5 6 7 8 # PostgreSQL connections watch 'psql -c "SELECT count(*) FROM pg_stat_activity"' # MySQL process list watch 'mysql -e "SHOW PROCESSLIST"' # Table row count watch 'psql -c "SELECT count(*) FROM users"' Quoting and Complex Commands For commands with pipes or special characters, quote the entire command: ...

htop is an interactive process viewer — a better top. It shows CPU, memory, running processes, and lets you kill or renice processes without typing PIDs. Installation 1 2 3 4 5 6 7 8 # Debian/Ubuntu sudo apt install htop # RHEL/CentOS sudo yum install htop # macOS brew install htop The Interface M S e w 1 5 1 2 3 4 m p P 2 6 [ [ I 3 7 | D 4 8 [ [ [ [ | U w p S w o E w s R - t d g a r t e a s P R 2 2 I 0 0 N I 0 0 1 V 0 5 I 2 1 R 4 2 4 T M M . 2 1 2 1 G 0 R 5 2 / K E 6 8 2 3 1 2 1 / S M M 5 3 8 8 5 2 . . . . . . 3 7 2 9 6 0 1 % % % % G 0 S 2 8 ] ] ] ] ] G H M M ] R S S S T L U 4 1 a o p C 5 2 s a t P . . k d i U 2 3 s m % : a e v : M 1 0 1 e E . . 4 r 4 M 6 8 2 a % , g d e a 1 3 : y 2 5 1 s T : : 2 1 , I 3 4 . M 4 3 t 2 0 E . . h 4 3 + 5 2 r : 6 1 ; 0 2 . 2 C n p 2 9 : o g o 8 1 m i s r 5 m n t u 0 a x g n . n : r n 8 d e i 7 w s n o : g r k w e r r i t e r Top Section CPU bars: Usage per core (user, system, nice, IRQ, etc.) Memory bar: Used/total RAM Swap bar: Used/total swap Tasks: Process count, threads, running processes Load average: 1, 5, 15 minute averages Uptime: System uptime Process Columns PID: Process ID USER: Owner PRI: Priority NI: Nice value (-20 to 19, lower = higher priority) VIRT: Virtual memory RES: Resident (physical) memory SHR: Shared memory S: State (R=running, S=sleeping, Z=zombie, D=disk wait) CPU%: CPU usage MEM%: Memory usage TIME+: Total CPU time Command: Process command Navigation ↑ P H S u / g o p ↓ U m a p e c / / e P E g n D d n M P F T F S F o a i a i e i v g r g l a l e e s t r t t p e c e u u / r r h r p p l o / / a c b d d s e y o o t s w w s u n n p s r e o r c e s s Function Keys F F F F F F F F F F 1 2 3 4 5 6 7 8 9 1 0 H S S F T S N N K Q e e e i r o i i i u l t a l e r c c l i p u r t e t e e l t p c e h r v b - + p ( i y r c e ( ( o o w c h l c n o i o e f l g w s i u h e s g m e r u n r r p a p r t r i i i o o o r n r i ) i t t y y ) ) Sorting F P M T I 6 O S S S S S I p o o o o o n e r r r r r v n t t t t t e r s b b b b b t o y y y y y r s t C M T n p o P E I e r r m U M M x e t e % % E t v n i o u c o r o u d l s e u r m c n o l u m n Process Actions Kill a Process 1 2 3 4 . . . . N P S - - - P a r e r v e l 1 9 2 e i s e 5 s g s c S S s a t S I I t F I G G E e 9 s G K I n i T I N t t g E L T e o o n R L r r a M ( p l ( i r k : ( f n o ) g o t c r r e e a c r s c e r s e ) u f p u t l ) ) Kill Multiple Processes 1 2 3 . . . S F A p 9 l a l c t e o t a t k g o i g l e t l d a g p r p o r c o e c s e s s e s s e s r e c e i v e s i g n a l Change Priority (Nice) F F 7 8 D I e n c c r r e e a a s s e e n n i i c c e e ( ( h l i o g w h e e r r p p r r i i o o r r i i t t y y ) , n e e d s r o o t ) Tree View Press F5 to toggle tree view — shows parent/child relationships: ...