You wrote what looks like perfectly reasonable Python code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import subprocess

proc = subprocess.Popen(
    ["some-command"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

# Read the output
output = proc.stdout.read()
errors = proc.stderr.read()
proc.wait()

And it hangs. Forever. No error, no timeout, just… nothing.

Welcome to the pipe buffer deadlock, one of Python’s most frustrating subprocess gotchas.

Why This Happens

The problem is OS pipe buffers. When you create a subprocess with stdout=subprocess.PIPE, the OS creates a pipe with a fixed buffer size (typically 64KB on Linux).

Here’s the sequence that causes the hang:

  1. Your subprocess writes to stdout
  2. The output goes into the pipe buffer
  3. The buffer fills up (64KB)
  4. The subprocess blocks, waiting for someone to read the pipe
  5. Your Python code is still waiting on proc.stdout.read()
  6. Deadlock: subprocess waits for Python to read, Python waits for subprocess to finish

If your command produces more than 64KB of output, you’re stuck.

The Fix: Use communicate()

The communicate() method exists specifically to solve this problem:

1
2
3
4
5
6
7
8
9

import subprocess

proc = subprocess.Popen(
    ["some-command"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

stdout, stderr = proc.communicate()

communicate() reads from both stdout and stderr simultaneously, preventing the buffer from filling up. It returns when the process exits and all output has been collected.

With a Timeout

1
2
3
4
5

try:
    stdout, stderr = proc.communicate(timeout=30)
except subprocess.TimeoutExpired:
    proc.kill()
    stdout, stderr = proc.communicate()  # Collect remaining output

When communicate() Isn’t Enough

Sometimes you need to process output as it arrives—for progress bars, real-time logging, or very large outputs that shouldn’t live in memory.

Option 1: Read Line by Line

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

import subprocess

proc = subprocess.Popen(
    ["some-command"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
    bufsize=1  # Line buffered
)

for line in proc.stdout:
    print(f"Got: {line.strip()}")

proc.wait()

Warning: This still deadlocks if stderr fills up while you’re reading stdout. For safety, redirect stderr:

1
2
3
4
5
6

proc = subprocess.Popen(
    ["some-command"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # Merge stderr into stdout
    text=True
)

Option 2: Use Threading

If you need both streams separately in real-time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import subprocess
import threading
from queue import Queue

def reader(pipe, queue):
    for line in pipe:
        queue.put(line)
    pipe.close()

proc = subprocess.Popen(
    ["some-command"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True
)

q = Queue()
stdout_thread = threading.Thread(target=reader, args=(proc.stdout, q))
stderr_thread = threading.Thread(target=reader, args=(proc.stderr, q))

stdout_thread.start()
stderr_thread.start()

while stdout_thread.is_alive() or stderr_thread.is_alive() or not q.empty():
    try:
        line = q.get(timeout=0.1)
        print(line, end='')
    except:
        pass

proc.wait()

Option 3: Use asyncio (Python 3.8+)

The cleanest solution for async codebases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import asyncio

async def run_command():
    proc = await asyncio.create_subprocess_exec(
        "some-command",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE
    )
    
    stdout, stderr = await proc.communicate()
    return stdout.decode()

result = asyncio.run(run_command())

Quick Reference: Which Method to Use

SituationSolution
Simple command, get all outputcommunicate()
Need timeoutcommunicate(timeout=N)
Process output line by linefor line in proc.stdout + merge stderr
Need both streams in real-timeThreading with queues
Already using asyncioasyncio.create_subprocess_exec()
Output too large for memoryStream to file, then process

The subprocess.run() Shortcut

For simple cases, skip Popen entirely:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import subprocess

result = subprocess.run(
    ["some-command"],
    capture_output=True,
    text=True,
    timeout=30
)

print(result.stdout)
print(result.stderr)
print(f"Exit code: {result.returncode}")

subprocess.run() uses communicate() internally, so it handles the buffer issue automatically.

Debugging Tips

If you’re still seeing hangs:

  1. Check for input: Does your command expect stdin? Pass stdin=subprocess.DEVNULL if not.

  2. Test output size: Run the command manually and check output size with command | wc -c.

  3. Add logging: Print before and after each subprocess operation to find where it sticks.

  4. Check for prompts: Some commands prompt for confirmation. Use yes | prefix or pass appropriate flags.

1
2
3
4
5
6
7

# Command that might prompt
proc = subprocess.Popen(
    ["rm", "-i", "file.txt"],
    stdin=subprocess.DEVNULL,  # Prevents waiting for input
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

Summary

The subprocess deadlock happens because OS pipe buffers are finite. When they fill up, everything blocks. The fix is almost always communicate()—it handles the complexity of reading from multiple pipes simultaneously.

Save yourself the debugging headache: default to subprocess.run() with capture_output=True for simple cases, and reach for Popen + communicate() only when you need more control.