Skip to content
md ~/writing/python-subprocess-tuzaklari-ffmpeg
All posts

· 6 min read

Python subprocess Pitfalls: Running ffmpeg Safely Without Hanging

Python subprocess Pitfalls: Running ffmpeg Safely Without Hanging

Running another program from Python looks like one line: subprocess.run("ffmpeg -i video.mkv video.mp4", shell=True). Once the program runs for a long time, its progress has to be read, the user picks the file name, and the same program runs hundreds of times, every part of that line becomes a separate problem. This post is about the Python subprocess pitfalls I ran into while running ffmpeg in my desktop audio and video converter: commands going through a shell, an unread pipe freezing the program, pipes left open, and file names starting with a dash.

What the app does

The user picks one or more files, chooses a target format (MP4, MP3, WebM…) and a quality. For each file the app first inspects it with ffprobe, then converts it with ffmpeg. During conversion it shows a progress bar and the time left, and the user can cancel at any moment. The UI is Tkinter; conversion runs on a separate thread so the window does not freeze.

Pitfall 1: handing the command to a shell as text

The shortest route is to write the command as one string and run it with shell=True. Then the string is interpreted by cmd.exe on Windows or sh on Linux. The file name sits inside that string, so the shell can read special characters in it as commands. A name like a"; del *.* & ".mp4 closes the quote and slips a new command in. The user picks the file, but they may not have named it: it may have been downloaded or sent by someone.

The fix is to skip the shell entirely and pass the command as a list. Each item goes to the program as exactly one argument; spaces, quotes or & inside it change nothing:

# Paths are always absolute: a file name starting with "-" must not look like an option
command = [ffmpeg, "-hide_banner", "-nostdin", "-loglevel", "error", "-i", str(Path(source).resolve())]
# ...
return command + ["-progress", "pipe:1", "-nostats", "-y", str(Path(target).resolve())]

Two more small protections are in that list:

  • Absolute paths: if a file is named like -y.mp4, ffmpeg may take it for an option. Path(...).resolve() always gives a full path starting with C:\... or /..., so no argument starts with a dash.
  • -nostdin: by default ffmpeg listens for keyboard commands (such as q to stop). It should not wait for or try to read input while running in the background, so that is off. The process is also started with stdin=subprocess.DEVNULL.

Pitfall 2: an unread pipe freezes the program

To show progress I use ffmpeg's -progress pipe:1 option. ffmpeg writes lines like these to standard output (stdout):

out_time_us=1500000
progress=continue

The app reads them and computes the percentage. Error messages go to standard error (stderr). To capture both, both are connected to pipes:

self.process = subprocess.Popen(
    self.command, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    text=True, encoding="utf-8", errors="replace", creationflags=NO_WINDOW,
)

There is a nasty trap here. Every pipe has a limited buffer in the operating system. If the main program reads only stdout line by line and ffmpeg writes a lot to stderr meanwhile, the stderr buffer fills up. ffmpeg waits for room to write its next line, but the main program never reads stderr, so the buffer never drains. ffmpeg cannot write to stdout either, and the main program waits for something on stdout. Each waits for the other: a deadlock. The progress bar stops somewhere and the app waits forever.

With -loglevel error ffmpeg normally writes very little to stderr. But on a damaged file it can write an error line for every frame, so the buffer can fill at exactly the worst moment. I reproduced it with a small experiment on my own machine: a child process that writes 200,000 lines to stderr was still stuck after 5 seconds when stderr was not read, and finished in half a second when it was.

The fix is to keep reading stderr on a second thread. Only the last 20 lines are kept so memory does not grow; that is enough to understand the error:

self.stderr = collections.deque(maxlen=20)  # only the last lines are kept

def _read_stderr(self):
    for line in self.process.stderr:
        self.stderr.append(line.rstrip())

# If the error output is not read separately, the buffer fills up and ffmpeg hangs
reader = threading.Thread(target=self._read_stderr, daemon=True)
reader.start()

Short commands where only the result matters do not have this problem. For a command like ffprobe, subprocess.run(..., capture_output=True) reads both pipes at the same time by itself; that is what I used there, with a timeout:

result = subprocess.run(
    command, capture_output=True, text=True, encoding="utf-8", errors="replace",
    timeout=PROBE_TIMEOUT_SECONDS, creationflags=NO_WINDOW,
)

The rule: when output is read line by line with Popen, no pipe may be left unread.

Pitfall 3: pipes left open

In the first version, two pipes stayed open after every conversion and file handles piled up. Pipes opened by Popen do not close when the process ends; they stay open until the garbage collector removes the Python object. You don't notice it with one conversion, but over a queue of hundreds of files they accumulate towards the operating system's limit.

The fix is two lines:

self.process.wait()
reader.join(timeout=2)
self.process.stdout.close()  # if the pipes are not closed, each conversion leaves a file handle open
self.process.stderr.close()

The order matters: first wait for the process to end, then for the thread reading stderr to finish, and only then close the pipes. A pipe closed in the other order breaks the thread that is still reading it.

Pitfall 4: cancelling and half-written files

When the user presses "Stop", the process is stopped with terminate(). But the half-written output file stays on disk, and the user may think it is a working video. So on cancel or error the target file is deleted:

if self.cancelled:
    self._remove_partial_output()
    return "cancelled"
if self.process.returncode != 0:
    self._remove_partial_output()
    return classify_error(self.stderr)

There is also a timing detail: if the user cancels right as the process is starting, the process object does not exist yet. So the flag is checked again as soon as the process starts:

if self.cancelled:  # cancel was pressed just as the process was starting
    self.process.terminate()

On errors the user never sees ffmpeg's raw output. The last stderr lines are searched for known phrases such as "No space left on device" or "Permission denied" and turned into a clear message.

What I took away

  • A list instead of shell=True. Every piece of outside text, file names included, should go as exactly one argument. Arguments that could start with a dash should be turned into absolute paths.
  • Every pipe connected with Popen has to be read. If one is not, its buffer fills and the two processes wait for each other. For short commands, run(capture_output=True) handles this itself.
  • Wait for the process, close the pipes. In a long-running app, small leaks add up.
  • When an external program stops, clean up after it: delete the half-written file and never show the raw error to the user.