Sysadmin

When a server is slow and people are yelling, you need a systematic approach. Here’s what to run in the first five minutes. The Checklist 1 2 3 4 5 6 7 8 uptime dmesg | tail vmstat 1 5 mpstat -P ALL 1 3 pidstat 1 3 iostat -xz 1 3 free -h sar -n DEV 1 3 Let’s break down what each tells you. 1. uptime 1 2 $ uptime 16:30:01 up 45 days, 3:22, 2 users, load average: 8.42, 6.31, 5.12 Load averages: 1-minute, 5-minute, 15-minute. ...

If you’re still typing ssh -i ~/.ssh/mykey.pem ec2-user@ec2-54-123-45-67.compute-1.amazonaws.com, you’re working too hard. SSH config transforms verbose commands into simple ssh prod invocations. The Basics Create or edit ~/.ssh/config: H o s t H U I p o s d r s e e o t r n d N t a e i m c t e 2 y - F 5 u i 4 s l . e e 1 r 2 ~ 3 / . . 4 s 5 s . h 6 / 7 p r o d - k e y . p e m Now ssh prod connects with the right key and user. No more remembering IP addresses. ...

Understanding processes is fundamental to Linux troubleshooting. These tools and techniques will help you find what’s running, what’s stuck, and what needs to die. Viewing Processes ps - Process Snapshot 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # All processes (BSD style) ps aux # All processes (Unix style) ps -ef # Process tree ps auxf # Specific columns ps -eo pid,ppid,user,%cpu,%mem,stat,cmd # Find specific process ps aux | grep nginx # By exact name (no grep needed) ps -C nginx # By user ps -u www-data Understanding ps Output U r w S o w E o w R t - d a t a 1 P 2 I 3 D 1 4 % C 0 2 P . . U 0 5 % M 0 1 E . . M 1 2 1 4 6 5 9 6 V 9 7 S 3 8 Z 6 9 1 9 3 8 R 2 7 S 5 6 S 6 5 T ? ? T Y S S S T s l A T S F 1 T e 0 A b : R 2 0 T 4 0 T 0 5 I : : M 0 2 E 3 3 C / n O s g M b i M i n A n x N / : D i n w i o t r k e r PID: Process ID %CPU: CPU usage %MEM: Memory usage VSZ: Virtual memory size RSS: Resident set size (actual RAM) STAT: Process state TIME: CPU time consumed Process States (STAT) R S D Z T N s l R S S Z S H L S M F u l l o t i o e u o n e e m o g w s l r n e e b p h s t e i p p i p p i i g n i i e e p r o - r g n n d r i n t o g g i o h u o r l r n ( ( r i e e d i u i t a a n n t y d d p t i y e e r e n r d o r t c r e e u r s p r s t u i p b t l i e b ) l e , u s u a l l y I / O ) top - Real-time View 1 2 3 4 5 6 7 8 9 10 11 # Basic top # Sort by memory top -o %MEM # Specific user top -u www-data # Batch mode (for scripts) top -b -n 1 Inside top: ...

strace intercepts and records system calls made by a process. When a program hangs, crashes, or behaves mysteriously, strace reveals what it’s actually doing at the kernel level. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # Trace a command strace ls # Trace running process strace -p 1234 # Trace with timestamps strace -t ls # Trace with relative timestamps strace -r ls Filtering System Calls 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Only file operations strace -e trace=file ls # Only network operations strace -e trace=network curl example.com # Only process operations strace -e trace=process bash -c 'sleep 1' # Specific syscalls strace -e open,read,write cat file.txt # Exclude syscalls strace -e trace=!mmap ls Trace Categories 1 2 3 4 5 6 7 file # open, stat, chmod, etc. process # fork, exec, exit, etc. network # socket, connect, send, etc. signal # signal, kill, etc. ipc # shmget, semop, etc. desc # read, write, close, etc. memory # mmap, brk, etc. Output Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Write to file strace -o output.txt ls # Append to file strace -o output.txt -A ls # With timestamps (wall clock) strace -t ls # With microseconds strace -tt ls # Relative timestamps strace -r ls Following Children 1 2 3 4 5 6 # Follow forked processes strace -f bash -c 'ls; echo done' # Follow forks and write separate files strace -ff -o trace ls # Creates trace.1234, trace.1235, etc. String Output 1 2 3 4 5 # Show full strings (default truncates at 32 chars) strace -s 1000 cat file.txt # Show full strings for specific calls strace -e read -s 10000 cat file.txt Statistics 1 2 3 4 5 6 7 8 9 # Summary of syscalls strace -c ls # Sample output: # % time seconds usecs/call calls errors syscall # ------ ----------- ----------- --------- --------- ---------------- # 45.00 0.000045 45 1 execve # 25.00 0.000025 3 8 mmap # 15.00 0.000015 2 6 openat 1 2 3 # Summary with detailed trace strace -c -S time ls # Sort by time strace -c -S calls ls # Sort by call count Practical Examples Debug “File Not Found” 1 2 # See what files the program is trying to open strace -e openat ./myprogram 2>&1 | grep -i "no such file" Find Config File Locations 1 2 # See all files a program tries to read strace -e openat nginx -t 2>&1 | grep -E "openat.*O_RDONLY" Debug Connection Issues 1 2 3 4 5 # Watch network connections strace -e connect curl https://example.com # See DNS lookups strace -e socket,connect,sendto,recvfrom dig example.com Debug Hanging Process 1 2 3 4 5 6 7 # Attach to hung process strace -p $(pgrep hung-process) # Common findings: # - Waiting on read() = blocked on input # - Waiting on futex() = waiting for lock # - Waiting on poll/select = waiting for I/O Find Why Program is Slow 1 2 3 4 5 6 7 8 # Time each syscall strace -T ls # Shows time spent in each call: # openat(AT_FDCWD, ".", ...) = 3 <0.000015> # Summary to find slow operations strace -c -S time slow-program Debug Permission Issues 1 2 3 4 5 # See access denials strace -e openat,access ./program 2>&1 | grep -i denied # Sample output: # openat(AT_FDCWD, "/etc/secret", O_RDONLY) = -1 EACCES (Permission denied) Watch File I/O 1 2 3 4 5 # See all reads and writes strace -e read,write -s 100 cat file.txt # Count I/O operations strace -c -e read,write dd if=/dev/zero of=/dev/null bs=1M count=100 Debug Signal Handling 1 2 3 4 5 # Trace signals strace -e signal,kill ./program # See what signal killed a process strace -e trace=signal -p 1234 Find Library Loading Issues 1 2 3 4 5 6 # See shared library loading strace -e openat ./program 2>&1 | grep "\.so" # Common issues: # - Library not found # - Wrong library version loaded Advanced Usage Inject Faults 1 2 3 4 5 # Make open() fail with ENOENT strace -e fault=openat:error=ENOENT ls # Fail every 3rd call strace -e fault=read:error=EIO:when=3 cat file.txt Decode Arguments 1 2 3 4 5 # Decode socket addresses strace -yy curl example.com # Decode file descriptors strace -y cat file.txt Trace Specific Syscall Return 1 2 3 4 5 # Only show failed syscalls strace -Z ls /nonexistent # Show syscalls that return specific value strace -e status=failed ls /nonexistent Reading strace Output o │ │ │ │ │ └ p ─ e ─ n a S t y ( s A t T │ │ │ │ └ e _ ─ m F ─ D c C D a W i l D r l , e c n " t a / o m e r e t y c │ │ │ └ / ─ f p ─ d a s P ( s a A w t T d h _ " n F , a D m C O e W _ D R │ │ └ a D ─ r = O ─ g N u c L F m u Y l e r ) a n r g t e = s n t 3 │ └ d ─ i ─ r ) R e t u r n v a l u e ( f d 3 ) Common Return Values = = = = = = = 0 N - - - - - 1 1 1 1 1 E E E E E N A A E I O C G X N E C A I T N E I S R T S N T S F N P R F I u i o e e i n c l r s l t c e s m o e e e u i u r s d c s r e r s e h s c x u s i e i p ( c f o s t f r i n t t e o i l e s d r p e d m t e p s m o o n o y a r r i r s n e a t y o d d r e r i i m c r l a c e y c l o c a l u t u l s n o n l ) t r a y v a i l a b l e Alternatives 1 2 3 4 5 6 7 8 # ltrace - trace library calls (not syscalls) ltrace ./program # perf trace - lower overhead perf trace ls # bpftrace - more powerful, requires setup bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s\n", str(args->filename)); }' Performance Note strace adds significant overhead — programs run much slower when traced. For production debugging: ...

lsof (list open files) shows which processes have which files open. Since Unix treats everything as a file — including network connections, devices, and pipes — lsof is incredibly powerful for system debugging. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # List all open files (overwhelming) lsof # List open files for specific file lsof /var/log/syslog # List open files for user lsof -u username # List open files for process lsof -p 1234 Find What’s Using a File 1 2 3 4 5 6 7 8 # Who has this file open? lsof /path/to/file # Who has files open in this directory? lsof +D /var/log # Recursive directory search lsof +D /var/log/ “Device or resource busy” Debugging 1 2 3 4 5 # Can't unmount? Find what's using it lsof +D /mnt/usb # Find open files on filesystem lsof /dev/sda1 Find What’s Using a Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # What's on port 8080? lsof -i :8080 # What's on port 80 or 443? lsof -i :80 -i :443 # All network connections lsof -i # TCP only lsof -i TCP # UDP only lsof -i UDP Specific Protocol and Port 1 2 3 4 5 6 7 8 # TCP port 22 lsof -i TCP:22 # UDP port 53 lsof -i UDP:53 # Port range lsof -i :1-1024 Filter by Process 1 2 3 4 5 6 7 8 9 10 11 # By PID lsof -p 1234 # By process name lsof -c nginx # By multiple process names lsof -c nginx -c apache # Exclude process lsof -c ^nginx Filter by User 1 2 3 4 5 6 7 8 9 10 11 # By username lsof -u www-data # By UID lsof -u 1000 # Multiple users lsof -u user1,user2 # Exclude user lsof -u ^root Network Connections List All Connections 1 2 3 4 5 6 7 8 # All internet connections lsof -i # Established connections only lsof -i | grep ESTABLISHED # Listening sockets lsof -i | grep LISTEN Connections to Specific Host 1 2 3 4 5 6 7 8 # Connections to specific IP lsof -i @192.168.1.100 # Connections to hostname lsof -i @example.com # Connections to specific host and port lsof -i @192.168.1.100:22 IPv4 vs IPv6 1 2 3 4 5 # IPv4 only lsof -i 4 # IPv6 only lsof -i 6 Output Formatting 1 2 3 4 5 6 7 8 9 10 # Terse output (parseable) lsof -t /path/to/file # Returns just PIDs # No header lsof +D /var/log | tail -n +2 # Specific fields lsof -F p /path/to/file # PID only lsof -F c /path/to/file # Command only Combining Filters By default, filters are OR’d. Use -a to AND them: ...

Unix permissions determine who can do what with files and directories. Understanding them is fundamental to system security and avoiding “Permission denied” errors. The Permission Model Every file has three permission types for three user classes: Permission Types: r (read) = 4 w (write) = 2 x (execute) = 1 User Classes: u (user/owner) g (group) o (others) Reading Permissions 1 2 ls -l file.txt # -rw-r--r-- 1 user group 1234 Feb 25 12:00 file.txt Breaking down -rw-r--r--: ...

htop is an interactive process viewer — a better top. It shows CPU, memory, running processes, and lets you kill or renice processes without typing PIDs. Installation 1 2 3 4 5 6 7 8 # Debian/Ubuntu sudo apt install htop # RHEL/CentOS sudo yum install htop # macOS brew install htop The Interface M S e w 1 5 1 2 3 4 m p P 2 6 [ [ I 3 7 | D 4 8 [ [ [ [ | U w p S w o E w s R - t d g a r t e a s P R 2 2 I 0 0 N I 0 0 1 V 0 5 I 2 1 R 4 2 4 T M M . 2 1 2 1 G 0 R 5 2 / K E 6 8 2 3 1 2 1 / S M M 5 3 8 8 5 2 . . . . . . 3 7 2 9 6 0 1 % % % % G 0 S 2 8 ] ] ] ] ] G H M M ] R S S S T L U 4 1 a o p C 5 2 s a t P . . k d i U 2 3 s m % : a e v : M 1 0 1 e E . . 4 r 4 M 6 8 2 a % , g d e a 1 3 : y 2 5 1 s T : : 2 1 , I 3 4 . M 4 3 t 2 0 E . . h 4 3 + 5 2 r : 6 1 ; 0 2 . 2 C n p 2 9 : o g o 8 1 m i s r 5 m n t u 0 a x g n . n : r n 8 d e i 7 w s n o : g r k w e r r i t e r Top Section CPU bars: Usage per core (user, system, nice, IRQ, etc.) Memory bar: Used/total RAM Swap bar: Used/total swap Tasks: Process count, threads, running processes Load average: 1, 5, 15 minute averages Uptime: System uptime Process Columns PID: Process ID USER: Owner PRI: Priority NI: Nice value (-20 to 19, lower = higher priority) VIRT: Virtual memory RES: Resident (physical) memory SHR: Shared memory S: State (R=running, S=sleeping, Z=zombie, D=disk wait) CPU%: CPU usage MEM%: Memory usage TIME+: Total CPU time Command: Process command Navigation ↑ P H S u / g o p ↓ U m a p e c / / e P E g n D d n M P F T F S F o a i a i e i v g r g l a l e e s t r t t p e c e u u / r r h r p p l o / / a c b d d s e y o o t s w w s u n n p s r e o r c e s s Function Keys F F F F F F F F F F 1 2 3 4 5 6 7 8 9 1 0 H S S F T S N N K Q e e e i r o i i i u l t a l e r c c l i p u r t e t e e l t p c e h r v b - + p ( i y r c e ( ( o o w c h l c n o i o e f l g w s i u h e s g m e r u n r r p a p r t r i i i o o o r n r i ) i t t y y ) ) Sorting F P M T I 6 O S S S S S I p o o o o o n e r r r r r v n t t t t t e r s b b b b b t o y y y y y r s t C M T n p o P E I e r r m U M M x e t e % % E t v n i o u c o r o u d l s e u r m c n o l u m n Process Actions Kill a Process 1 2 3 4 . . . . N P S - - - P a r e r v e l 1 9 2 e i s e 5 s g s c S S s a t S I I t F I G G E e 9 s G K I n i T I N t t g E L T e o o n R L r r a M ( p l ( i r k : ( f n o ) g o t c r r e e a c r s c e r s e ) u f p u t l ) ) Kill Multiple Processes 1 2 3 . . . S F A p 9 l a l c t e o t a t k g o i g l e t l d a g p r p o r c o e c s e s s e s s e s r e c e i v e s i g n a l Change Priority (Nice) F F 7 8 D I e n c c r r e e a a s s e e n n i i c c e e ( ( h l i o g w h e e r r p p r r i i o o r r i i t t y y ) , n e e d s r o o t ) Tree View Press F5 to toggle tree view — shows parent/child relationships: ...