Python 3 Subprocess Examples

Last updated:
Table of Contents

All examples use Python 3.5 or later (unless noted) and assume you're running Linux or a unix-based OS.

All examples can be found on this Jupyter notebook

call() example

Use run() instead on Python v3.5+

With suprocess.call() you pass an array of commands and parameters.

import subprocess

subprocess.call(["ls", "-lha"])
# returns 0 (the return code)

call() example using shell=True

Use run() instead on Python v3.5+

If shell=True, the command string is interpreted as a raw shell command.

Using shell=True may expose you to code injection if you use user input to build the command string.

subprocess.call("ls -lha", shell=True)
# returns 0 (the return code)
Twitter Linkedin YC Hacker News Reddit

Run command and capture output

Using universal_newlines=True converts the output to a string instead of a byte array.

  • Python version 2.7 -> 3.4

    import subprocess
    
    # errors in the created process are raised here too
    output = subprocess.check_output(["ls","-lha"],universal_newlines=True)
    
    output
    # total 20K
    # drwxrwxr-x  3 felipe felipe 4,0K Nov  4 15:28 .
    # drwxrwxr-x 39 felipe felipe 4,0K Nov  3 18:31 ..
    # drwxrwxr-x  2 felipe felipe 4,0K Nov  3 19:32 .ipynb_checkpoints
    # -rw-rw-r--  1 felipe felipe 5,5K Nov  4 15:28 main.ipynb
    
  • Python version 3.5+

    import subprocess
    
    # run() returns a CompletedProcess object if it was successful
    # errors in the created process are raised here too
    process = subprocess.run(['ls','-lha'], check=True, stdout=subprocess.PIPE, universal_newlines=True)
    output = process.stdout
    
    output
    # total 20K
    # drwxrwxr-x  3 felipe felipe 4,0K Nov  4 15:28 .
    # drwxrwxr-x 39 felipe felipe 4,0K Nov  3 18:31 ..
    # drwxrwxr-x  2 felipe felipe 4,0K Nov  3 19:32 .ipynb_checkpoints
    # -rw-rw-r--  1 felipe felipe 5,5K Nov  4 15:28 main.ipynb
    

run() example: run command

run() behaves mostly the same way as call() and you should use it instead of call() for version 3.5 onwards.

See section call() vs run() below

import subprocess

cp = subprocess.run(["ls","-lha"])

cp
# CompletedProcess(args=['ls', '-lha'], returncode=0)

run() example: using shell=True

As in the call() example, shell=True, the command string is interpreted as a raw shell command.

Again, Using shell=True may expose you to code injection if you use user input to build the command string.

import subprocess

cp = subprocess.run(["ls -lha"],shell=True)

cp
#CompletedProcess(args=['ls -lha'], returncode=0)

run() example: store output and error message in string

If the underlying process returns a nonzero exit code, you will not get an exception; the error message can be accessed via the stderr attribute in the CompletedProcess object.

  • case 1: process return 0 exit code

    import subprocess
    
    cp = subprocess.run(["ls","-lha"], universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    
    cp.stdout
    # total 20K
    # drwxrwxr-x  3 felipe felipe 4,0K Nov  4 15:28 .
    # drwxrwxr-x 39 felipe felipe 4,0K Nov  3 18:31 ..
    # drwxrwxr-x  2 felipe felipe 4,0K Nov  3 19:32 .ipynb_checkpoints
    # -rw-rw-r--  1 felipe felipe 5,5K Nov  4 15:28 main.ipynb
    cp.stderr
    # '' (empty string)
    cp.returncode
    # 0
    
  • case 2: process returns nonzero exit code

    import subprocess
    
    cp = subprocess.run(["ls","foo bar"], universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    
    cp.output
    # '' (empty string)
    cp.stderr
    # ls: cannot access 'foo bar': No such file or directory
    cp.returncode
    # 2
    
  • case 3: other OS-level errors

    this case will throw an exception no matter what. For example, if you call an executable that doesn't exist. This throws an exception because it wasn't that the subprocess had an error - it never got created in the first place.

    import subprocess
    
    try:
        cp = subprocess.run(["xxxx","foo bar"], universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    except FileNotFoundError as e:
        print(e)
        # [Errno 2] No such file or directory: 'xxxx'
    

Popen example: run command and get return code

subprocess.Popen() is used for more complex examples where you need. See Popen() vs call() vs run()

This causes the python program to block until the subprocess returns.

from subprocess import Popen

p = Popen(["ls","-lha"])
p.wait()
# 0

Popen example: Store the output and error messages in a string

import subprocess
from subprocess import Popen

p = Popen(["ls","-lha"], stdout=subprocess.PIPE, stderr= subprocess.PIPE, universal_newlines=True)

output, errors = p.communicate()

output
# total 20K
# drwxrwxr-x  3 felipe felipe 4,0K Nov  4 15:28 .
# drwxrwxr-x 39 felipe felipe 4,0K Nov  3 18:31 ..
# drwxrwxr-x  2 felipe felipe 4,0K Nov  3 19:32 .ipynb_checkpoints
# -rw-rw-r--  1 felipe felipe 5,5K Nov  4 15:28 main.ipynb

errors
# '' (empty string)

Popen example: Redirect output to file

import subprocess
from subprocess import Popen

path_to_output_file = '/tmp/myoutput.txt'

myoutput = open(path_to_output_file,'w+')

p = Popen(["ls","-lha"], stdout=myoutput, stderr= subprocess.PIPE, universal_newlines=True)

output, errors = p.communicate()

output
# there's nothing here because we didn't send stdout to subprocess.PIPE

errors
# '' empty string

# stdout has been written to this file
with open(path_to_output_file,"r") as f:
    print(f.read())

# total 20K
# drwxrwxr-x  3 felipe felipe 4,0K Nov  4 17:00 .
# drwxrwxr-x 39 felipe felipe 4,0K Nov  3 18:31 ..
# drwxrwxr-x  2 felipe felipe 4,0K Nov  3 19:32 .ipynb_checkpoints
# -rw-rw-r--  1 felipe felipe 7,7K Nov  4 17:00 main.ipynb

Popen example: Redirect output and errors to the same file

import subprocess
from subprocess import Popen

path_to_output_file = '/tmp/myoutput.txt'

myoutput = open(path_to_output_file,'w+')

# file 'foo bar' doesn't exist
p = Popen(["ls","foo bar"], stdout=myoutput, stderr=myoutput, universal_newlines=True)

output, errors = p.communicate()

output
# there's nothing here because we didn't send stdout to subprocess.PIPE

errors
# there's nothing here either

# stdout and stderr have been written to this file
with open(path_to_output_file,"r") as f:
    print(f.read())

# ls: cannot access 'foo bar': No such file or directory

Popen example: Run command in the background

By default, calls to Popen() spawn a subprocess in the background and don't wait for it to terminate (unless you use wait() on the Popen object).

Pipe commands together

Use Popen:

from subprocess import Popen,PIPE

# this is equivalent to ls -lha | grep "foo bar"
p1 = Popen(["ls","-lha"], stdout=PIPE)
p2 = Popen(["grep", "foo bar"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()

output = p2.communicate()[0]

Wait for command to terminate, asynchronously

Use asyncio and

Method asyncio.create_subprocess_exec() works much the same way as Popen() but calling wait() and communicate() on the returned objects does not block the processor, so the Python interpreter can be used in other things while the external subprocess doesn't return.

Python 3.6+ is needed here

import asyncio

proc = await asyncio.create_subprocess_exec(
    'ls','-lha',
    stdout=asyncio.subprocess.PIPE,
    stderr=asyncio.subprocess.PIPE)


# if proc takes very long to complete, the CPUs are free to use cycles for 
# other processes
stdout, stderr = await proc.communicate()

proc.returncode
# 0

# must call decode because stdout is a bytes object
stdout.decode()
# total 24K
# drwxrwxr-x  3 felipe felipe 4,0K Nov  4 17:52 .
# drwxrwxr-x 39 felipe felipe 4,0K Nov  3 18:31 ..
# drwxrwxr-x  2 felipe felipe 4,0K Nov  3 19:32 .ipynb_checkpoints
# -rw-rw-r--  1 felipe felipe  11K Nov  4 17:52 main.ipynb

stderr.decode()
# ''  empty string    

call() vs run()

As of Python version 3.5,run() should be used instead of call().

  • run() returns a CompletedProcess object instead of the process return code.

  • other functions like check_call() and check_output() can all be replaced with run().

Twitter Linkedin YC Hacker News Reddit

Popen vs run() and call()

call() and run() are convenience functions and should be used for simpler cases.

Popen() is much more powerful and handles all cases, not just simple ones.

Dialogue & Discussion