Logging

Your logs are probably useless. Not because you’re not logging, but because you’re logging wrong. Here’s how to make logs that actually help when things break. The Problem with Unstructured Logs [ [ [ 2 2 2 0 0 0 2 2 2 4 4 4 - - - 0 0 0 3 3 3 - - - 1 1 1 2 2 2 1 1 1 0 0 0 : : : 2 2 2 3 3 3 : : : 4 4 4 5 6 7 ] ] ] I E P N R r F R o O O c : R e : s U s s S i e o n r m g e l t o o h r g i d g n e e g r d w f i e o n n r t u w s r e o r n g 1 2 3 4 5 Try answering these questions: ...

You’ve seen these logs: 2 2 2 0 0 0 2 2 2 6 6 6 - - - 0 0 0 3 3 3 - - - 1 1 1 0 0 0 0 0 0 7 7 7 : : : 0 0 0 0 0 0 : : : 0 0 0 0 1 1 I E I N R N F R F O O O R P R r S e o o t c m r e e y s t i s h n i i g n n . g g . . r w e e q n u t e s w t r o n g Good luck debugging that at 3 AM. Which request? What went wrong? Retrying what? ...

Your logs are lying to you. Not because they’re wrong, but because they’re formatted for humans who will never read them. That stack trace you carefully formatted? It’ll be searched by a machine. Those helpful debug messages? They’ll be filtered by a regex that breaks on the first edge case. The log line that would have saved you three hours of debugging? Buried in 10GB of unstructured text. Structured logging fixes this. Here’s how to do it without making your codebase worse. ...

Everyone logs. Few log well. The difference between “we have logs” and “we can debug with logs” comes down to discipline in what you capture, how you structure it, and where you send it. The Logging Hierarchy Not all log levels are created equal. Use them intentionally: F E W I D T A R A N E R T R R F B A A O N O U C L R G E → → → → → → T S U N D E h o n o e x e m e r t t e x m a r a t p a i e p h e l l m p i c e e l n t o d l i g e p y c d e d a f r i v t a b a a e i i u t g r o l t i n b n e o o o d h n s s c , a t e a n m i . n b d i c n u l l N o t e e i e t d s n v t . t f e c h o o r o e M n . n i e i t a g s E n i p h . x n p t p p u T e r e k b h n o . e e e s d e c i u W p o h v c a s m e e t k e a , i e r r o u a t u n s n b s . o n p e u m i r a a e n o t l o g b . l n . l y e e N m o u e . f p e f . d s i n a t p t r e o n d t . i o n . The key insight: INFO should tell a story. If you read only INFO logs, you should understand what the application did. ...

When your application runs on 50 containers across 10 servers, SSH’ing into each one to grep logs doesn’t scale. Centralized logging gives you one place to search everything. The Log Aggregation Pipeline A p p s f s l t i y i d l s c ↓ o e l a u s o t t g i o n s → F L V C l o e o u g c l e s t l n t o e ↓ t a r c d s t h o r s → E L S S l o 3 t a k o s i r ↓ t a i g c e s e → a r S c e h a r c K G C h i r L / ↓ b a I V a f i n a s a n u a a l i z a t i o n Stack Options ELK (Elasticsearch, Logstash, Kibana) The classic choice. Powerful but resource-hungry. ...

When your service goes down at 3 AM, you need answers fast. Observability—the ability to understand what’s happening inside your systems from their external outputs—is what separates a 5-minute fix from a 3-hour nightmare. The three pillars of observability are logs, metrics, and traces. Each tells a different part of the story. Logs: The Narrative Logs are discrete events. They tell you what happened in human-readable terms. 1 2 3 4 5 6 7 8 9 { "timestamp": "2026-03-03T12:34:56Z", "level": "error", "service": "payment-api", "message": "Payment processing failed", "user_id": "12345", "error_code": "CARD_DECLINED", "request_id": "abc-123" } Best Practices for Logging Structure your logs. JSON is your friend. Unstructured logs like Payment failed for user 12345 are hard to search and aggregate. ...

Logging seems simple until you’re debugging production at 2 AM, scrolling through millions of lines trying to find the one that matters. Good logging practices make that experience less painful. Here’s how to think about log levels. The Levels Most logging frameworks use these standard levels: D E B U G < I N F O < W A R N < E R R O R < F A T A L In production, you typically run at INFO or WARN. Lower levels include all higher levels (INFO includes WARN, ERROR, and FATAL). ...

Unstructured logs are technical debt. Structured logs are queryable, parseable, and actually useful when things break. The Problem # 2 2 2 0 0 0 U 2 2 2 n 6 6 6 s - - - t 0 0 0 r 2 2 2 u - - - c 2 2 2 t 8 8 8 u r 1 1 1 e 0 0 0 d : : : : 1 1 1 5 5 5 g : : : o 2 2 2 o 3 4 5 d I E I l N R N u F R F c O O O k R U R p s F e a e a q r r i u s l e i a e s n l d t g i c t c t e o o h m i l p p s o r l g o e g c t e e e d s d s i i n o n r f d 2 r e 3 o r 4 m m 1 s 1 2 9 3 2 4 . 5 1 : 6 8 c . o 1 n . n 1 e c t i o n t i m e o u t Regex hell when you need to extract user, IP, order ID, or duration. ...

Unstructured logs are a trap. They look simple until you need to find something. [ [ [ 2 2 2 0 0 0 2 2 2 6 6 6 - - - 0 0 0 2 2 2 - - - 2 2 2 7 7 7 0 0 0 5 5 5 : : : 3 3 3 0 0 0 : : : 1 1 1 5 6 7 ] ] ] I E W N R A F R R O O N R U H s F i e a g r i h l j e m o d e h m n t o @ o r e y x p a r u m o s p c a l e g e s e . s c d o o e m r t d e l e c o r t g e g 1 d e 2 : d 3 4 8 i 5 7 n : % f c r o o n m n e 1 c 9 t 2 i . o 1 n 6 8 t . i 1 m . e 5 o 0 u t Quick: find all login failures from a specific IP range in the last hour. Now try parsing the order ID from error messages. Hope you enjoy regex. ...

systemd’s journal collects logs from all services, the kernel, and system messages in one place. journalctl is your tool for searching, filtering, and following those logs. Basic Usage 1 2 3 4 5 6 7 8 9 10 11 # Show all logs (oldest first) journalctl # Show all logs (newest first) journalctl -r # Follow new entries (like tail -f) journalctl -f # Show only errors and above journalctl -p err Filter by Time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # Since boot journalctl -b # Previous boot journalctl -b -1 # Since specific time journalctl --since "2024-02-25 10:00:00" # Until specific time journalctl --until "2024-02-25 12:00:00" # Time range journalctl --since "1 hour ago" journalctl --since "2024-02-25" --until "2024-02-26" # Relative times journalctl --since "yesterday" journalctl --since "10 minutes ago" Filter by Unit (Service) 1 2 3 4 5 6 7 8 9 10 11 # Specific service journalctl -u nginx # Multiple services journalctl -u nginx -u php-fpm # Follow specific service journalctl -u nginx -f # Service since boot journalctl -u nginx -b Filter by Priority Priority levels (0=emergency to 7=debug): ...