Amazon S3: Common CLI Commands & Reference

Last updated:

WIP Alert This is a work in progress. Current information is correct but more content will probably be added in the future.

S3 is one of the most widely used AWS offerings. After installing awscli (see references for info) you can access S3 operations in two ways:

  • $ aws s3 <command> (for simple filesystem stuff like mv, cp and so on)

  • $ aws s3api <command> (for other operations)

Use --help to view

We will use "samplebucket" as an example bucket name throughout

Download a single file

cpis for copy; . stands for the current directory

$ aws s3 cp s3://samplebucket/somefolder/afile.txt .

Download a specific folder and all subfolders, recursively

Useful for downloading log folders

$ aws s3 cp s3://samplebucket/somefolder/ . --recursive

Delete a folder in a bucket

(along with any data within)

$ aws s3 rm s3://samplebucket/somefolder --recursive

Upload a single file to an S3 bucket

$ aws s3 cp /path/to/localfile s3://samplebucket/somefolder

Upload multiple files at the same time

See my post on linux find examples for more ways to use find

Say, for instance, you have multiple files that you want to upload to the same bucket. If they all have names

$ ls
aa ab ac ad ae
$ find . -name "x*" | xargs -I {} aws s3 cp {} s3://samplebucket/samplefolder

View stats about a bucket (total size, etc)

$ aws s3 ls s3://samplebucket/path/to/directory/ --summarize --human-readable --recursive
  • sample output (some rows ommited)
2015-08-24 23:11:03   40.0 MiB news.en-00098-of-00100
2015-08-24 23:11:04   39.9 MiB news.en-00099-of-00100
Total Objects: 100
Total Size: 3.9 GiB

Download just part of a large file from S3

For instance, download only the first 1MB (1 million bytes) from a file located under s3://samplebucket/path/to/file.csv

$ aws s3api get-object --range "bytes=0-1000000" --bucket "samplebucket" --key "path/to/file.csv" output

References

Dialogue & Discussion