3 thoughts on “How to Copy local files to S3 with AWS CLI” Benji April 26, 2018 at 10:28 am. What protocol is used when copying from local to an S3 bucket when using AWS CLI?

• Aws Cli Credentials
• Aws Cli Download Windows
• Aws Cli Download S3 Folders

After you have CLI installed on your system, you can begin using it to perform useful tasks for AWS. For a developer, that means being able to perform configuration, check status, and do other sorts of low-level tasks with the various AWS services. The information here helps you understand how you can use CLI to perform essential tasks with S3. Going through these exercises helps you better understand how S3 works, in addition to allowing you to perform development-required tasks.

Creating the aws utility configuration file

To use the aws utility to perform tasks using AWS CLI, you must create a configuration file. The configuration file contains a number of pieces of information, including both your public and secret keys. The following steps help you perform this configuration task:

1. Open a command prompt or terminal window.
2. Type aws configure and press Enter. You see a prompt asking for your public ke.
3. Type your public key string and press Enter.

   In most cases, you can copy and paste your key directly from the .csv file used to store it. The method you use depends on your operating system. For example, when working at the Windows command prompt, you right-click and choose Paste from the context menu. You see a prompt asking for your private key.

4. Type your private (secret) key string and press Enter.

   You see a prompt asking for the default region used to access data. The region you provide, such as us-west-2, should match the region you use when interacting with AWS from the consoles.

5. Type the region information and press Enter.

   The configuration routine asks for an output format. Choose one of the following options:

   • json: The default format outputs the data using the JavaScript Object Notation (JSON) technique, which relies on name/value pairs. An advantage of this format is that it works well for direct input with some languages, such as Python. Check out this basic JSON tutorial.
   • text: Outputs the data using simple text. The advantage of this approach is that no formatting is involved, so you can easily modify it to meet any need. However, the output can be a little hard to read.
   • table: Outputs the data using table-formatted text. The advantage of this approach is that the output is easily read directly at the command line.
6. Type the output format and press Enter.

   You return to the command prompt.

• Jul 12, 2018 - To Download using AWS S3 CLI: aws s3 cp s3://WholeBucket LocalFolder --recursive aws s3 cp s3://Bucket/Folder LocalFolder --recursive.
• The AWS Command Line Interface is a unified tool that provides a consistent interface for interacting with all parts of AWS. Install AWS CLI using command sudo pip install awscli and then follow below command to download entire S3 bucket.

The configuration command creates two new files for you. Both of these files appear in the .aws folder on your system. The precise location depends on the operating system you use. For example, on a Windows system, you generally find the files in the C:Users.aws folder. After you complete this task, the config file contains the region you want to use and the output format. However, you can add other entries as needed. The credentials file contains your public and private keys.

Obtaining S3 information

To ensure that your aws utility works as expected, you need to try a test access of AWS. Type aws s3 ls and press Enter. You begin with the aws utility, followed by the name of the service you want to access, which is s3. The ls command lists the content of an S3 object. Because you haven’t provided a specific location in S3, what you see as output is a listing of the S3 buckets you’ve created. Note that the output contains the execution date and time. The bucket name will match the name you provided.

You can try uploading a file to your bucket. To perform this task, you use the copy or cp command. The cp command requires that you provide a source location and a destination location. The source and destination can be a local folder or S3 bucket. Although you wouldn’t use this technique to perform a local copy, you can copy from a local folder to an S3 bucket, from an S3 bucket to a local folder, or between S3 buckets.

For example, to copy a file named colorblk.gif from a local folder named win to the S3 bucket, you would type something like aws s3 cp 'c:wincolorblk.gif' s3://aws4d.test-bucket/colorblk.gif and press Enter. You must provide a source and destination that match your setup. To ensure that the file is actually uploaded, you use the ls command again, but this time you add the bucket name.

I've got several large files sitting in my Linux hosted account that I need to upload to my S3 account. I don't want to download them first and then upload them into S3.

Is there any way I can 'upload' it via the Linux command line? Or can I access it via a website working with Lynx?

6 Answers

S3cmd does what you want. Uploading and downloading files, syncing directories and creating buckets.

S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. It is best suited for power users who are familiar with command line programs. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.

Amazon provides their own CLI tools now too.

From http://aws.amazon.com/cli/

Using familiar syntax, you can view the contents of your S3 buckets in a directory-based listing.

You can perform recursive uploads and downloads of multiple files in a single folder-level command. The AWS CLI will run these transfers in parallel for increased performance.

A sync command makes it easy to synchronize the contents of a local folder with a copy in an S3 bucket.

Documentation for file related commands is here.

Aws Cli Credentials

If you can't (perhaps you're on a shared host) or don't want to install extra tools, it is possible to just use bash, curl, and openssl.

Note that I modified this script from the one in the above link. I added the -L option because AWS may insert a redirect in there. The -L option will follow the redirect for you.

One other caveat. This won't work for files larger than 5GB. Those require a multi-part upload that would require a more complex script.

A POSIX-compliant shell script that requires openssl, curl and sed only; supporting AWS Signature Version 4, which is required for region eu-central-1 (Frankfurt) and recommended for the others:

Notice, the script will enable server-side

AES256 encryption by default.

Aws Cli Download Windows

Alternatively you can try https://github.com/minio/mc

mc provides minimal tools to work with Amazon S3 compatible cloud storage and filesystems. It has features like resumable uploads, progress bar, parallel copy. mc is written in Golang and released under Apache license v2.

I've found Python's AWS bindings in the boto package (pip install boto) to be helpful for uploading data to S3.

The following script can be called like: python script_name.py 'sub_bucket_name' '*.zip' where sub_bucket_name indicates the name of the directory in which the files should be stored in S3, and *.zip is a glob path designating one or more files to be uploaded:

Aws Cli Download S3 Folders

Not the answer you're looking for? Browse other questions tagged linuxcommand-lineuploadamazon-s3 or ask your own question.