Amazon S3 is an online file storage web service offered by Amazon Web Services. Amazon S3 provides its service through REST, SOAP or Bittorrent web services.

s3cmd is a command line tool written in the Python language to interact with the Amazon S3 service.

Until recently the version 1.5.0 of the s3cmd tool was in alpha phase and you could only get it from source or third party package repositories.

The version 1.5.0 has been released about 3 weeks ago and adds some very cool features, one of the new options in this new release is the support of multi-part uploads.

How do I install s3cmd?

There are a few different options, of which a couple is discussed on http://s3tools.org website.

  • If you are using OSX and are using Homebrew, you can do brew install s3cmd and that will pull and install version 1.5.0 of the tool.
  • If you are using Linux and are using Linuxbrew you can also do brew install s3cmd.
  • If you are not using either, then you can install the s3cmd by doing pip install -V s3cmd==1.5.0a3 (for some reason the PyPy repository still has an outdated release).

What is that new feature that was mentioned in the title?

Well, Amazon S3 has some throttling limitations when uploading large files, see here:

s3cmd put ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box s3://bucket-for-vagrant/trusty64/
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box  [1 of 1]
  16068608 of 659236638     2% in   22s   707.26 kB/s  failed
WARNING: Upload failed: /trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box  [1 of 1]
  54104064 of 659236638     8% in   62s   841.51 kB/s  failed
WARNING: Upload failed: /trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1.box  [1 of 1]
 188264448 of 659236638    28% in  486s   378.21 kB/s
 ^CSee ya!

With the new version of s3cmd, when you can add the new option to the put command in this way s3cmd put --continue-put, the upload, a large file, will be splitted into parts and the s3cmd tool will upload them individually to be joined on the server side (Amazon S3). The output of this new command looks like it can be seen below (I am truncating the logs for brevity purposes):

s3cmd put --continue-put -H ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box s3://bucket-for-vagrant/trusty64/
WARNING: Forwarding request to us-east-1
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box  [part 1 of 66, 15MB]
 15728640 of 15728640   100% in   20s   745.70 kB/s  done
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box  [part 2 of 66, 15MB]
 15728640 of 15728640   100% in   23s   652.57 kB/s  done
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box  [part 3 of 66, 15MB]
 15728640 of 15728640   100% in   22s   668.35 kB/s  done
ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box -> s3://bucket-for-vagrant/trusty64/ubuntu-trusty-64-1.2.1-virtualbox-puppet-3.7.4-1puppetlabs1-java8.box  [part 4 of 66, 15MB]
 15728640 of 15728640   100% in   21s   721.65 kB/s  done
 (...snip...)

So, if you already used s3cmd go upgrade to the latest version ASAP!