Working with Files & the Filesystem in Python: Reference and Examples
Last updated:- Write string to file
- Read whole file into string
- Read file line by line into string
- Read file line by line into list
- Check if file exists
- Relative to absolute path
- Relative to absolute path in Jupyter notebooks
- Get file name from file path (basename)
- Get path to file without the basename
- Get current working directory
- Get directory where this (current) file is located
- List files in directory
- For file in directory
- For file in directory, recursively
- Delete file
- Move/rename file
- Copy file
- Add directory to module Include Path
- Add directory to system PATH
All examples assume Python 3, unless otherwise stated
The full list of methods you can call on a file object can be found here: Python 3: io.IOBASE
Manipulating filesystem paths is a very common need in most if not all applications; most problems can be solved using either os.path or pathlib modules.
Pathlib is used in some examples here; it comes by default with Python 3 but you can also use it in Python 2 via Pip:
pip install pathlib
Write string to file
write
is just one method of class File
. See all here
# will be overwritten if it already exists!
with open("filename.txt","w") as f:
f.write("foo bar")
Read whole file into string
with open("filename.txt","r") as f:
contents = f.read()
Read file line by line into string
Without loading it all to memory (buffered reading)
This is easy with python:
with open("path/to/file") as f:
for line in f:
# process the line
You can also use f.readline()
to read a whole line (dangerous if the file has no line breaks) or f.read(size)
, which takes an argument indicating the maximum number of bytes to be read into memory.
Read file line by line into list
line_list = []
with open("path/to/file") as f:
for line in f:
# drop the newline at the end
line_list.append(line.rstrip('\n'))
Check if file exists
This method returns True
if the given path exists and it's a file.
If you just want to test if a path exists (be it a file, directory or even link), use os.path.exists()
instead.
import os.path
path_to_file = "/path/to/file"
if os.path.isfile(path_to_file):
# it exists, and it is a regular
# (i.e. not a directory) file
Relative to absolute path
Use os.path.abspath
. But remember that the relative path is calculated with respect to your current working directory.
Note that this will not warn you if you supply a path that resolves to a file/dir that doesn't exist
If you want the relative path to be calculated with respect to the current file location, don't forget to prepend __file__+"/"
to the relative path!
import os
# path is relative to the current WORKING directory!
abs_path = os.path.abspath("../../other-dir")
# path is relative to the directory where THIS file
# is located!
abs_path = os.path.abspath(__file__+"/../../other-dir")
Relative to absolute path in Jupyter notebooks
"NameError: name 'file' is not defined"
Use "__file__"
(in quotes) instead:
import os
# on a jupyter notebook
abs_path = os.path.abspath("__file__"+"/../../other-dir")
Get file name from file path (basename)
import os
print(os.path.basename("/path/to/some/file.txt"))
# "file.txt"
Get path to file without the basename
This is the inverse of the above. Note the trailing slash is not included.
Just use
import os
print(os.path.dirname("/path/to/some/file.txt"))
# "/path/to/some"
Get current working directory
The current working directory (or cwd) is the directory the user that first called the script was located.
import os
os.getcwd()
Get directory where this (current) file is located
Note that this is not at all the same thing as the current working directory (seen above)
import os
# __file__ points to the where the current file is located
os.path.dirname(os.path.realpath(__file__))
List files in directory
import os
os.listdir(path_to_directory)
For file in directory
import os
root_path = "/path/to/directory"
for file in os.listdir(root):
# myfile.txt
file_name = file
# /path/to/directory/myfile.txt
full_path_to_file = os.path.join(root,file)
For file in directory, recursively
To loop over all files recursively under some path (i.e. including subdirectories):
import os
# you may omit trailing slashes
# i.e. '/tmp' works too
path = '/tmp/'
for root, directories, filenames in os.walk(path):
for directory in directories:
directory_path = os.path.join(root, directory)
# do something with directory_path
for filename in filenames:
file_path = os.path.join(root,filename)
# do something with directory_path
Delete file
Template os.remove(<path-to-file>)
.
import os
# raises FileNotFoundError if file doesn't exist
os.remove("/path/to/file")
Move/rename file
Template: os.replace(<source-path>,<target-path>)
import os
# raises FileNotFoundError if source file doesn't exist
os.replace('/path/to/source/file','/path/to/new/location/file')
Copy file
Template: shutil.copy(<from>,<to>)
and shutil.copy2(<from>,<to>)
. Both methods are used the same way. See differences below.
Shutil is part of Python's standard library; there's no need to install anything.
Method | Copies file data | Copies file permissions | Copies file creation and modfication dates |
---|---|---|---|
shutil.copy | YES | YES | NO |
shutil.copy2 | YES | YES | YES |
Target path can be the full path to the new location
import shutil source_path = '/path/to/source/source-file' target_path = '/path/to/target/target-file' # raises FileNotFoundError if source file doesn't exist # target file is overwritten if it already exists! shutil.copy(source_path,target_path) # >>> '/path/to/target/target-file'
Target path can also be the path to a directory
import shutil source_path = '/path/to/my-file' target_path = '/path/to/another/directory/' # raises FileNotFoundError if source file doesn't exist # target file is overwritten if it already exists! shutil.copy(source_path,target_path) # >>> '/path/to/another/directory/my-file'
Add directory to module Include Path
Use sys.path.insert(0,"/absolute/path/to/directory")
Example: Suppose you have a directory structure that looks like this:
myproject/
├── helpers
│ └── mymodule.py
├── utils
│ └── myothermodule.py
└── stuff
└── file1.py
# stuff/file1.py
import os
import sys
# find out the absolute path to the directory
# you want to add to your path
other_dir = os.path.abspath(__file__+"/../")
sys.path.insert(0,other_dir)
# now you can import stuff like this
from helpers import mymodule
from utils import myothermodule
Add directory to system PATH
That is, the PATH
environment variable.
import os
# use os.pathsep so that this works under linux or windows
os.environ["PATH"] += os.pathsep + "/path/to/directory"