GreenQloud Community/Documentation: How-Tos and Tutorials/Storage API (S3 compatible)

Using s3cmd and s3fs with the Storage API (Centos 5.6)

Eikster
posted this on June 07, 2011 13:45

Here's how to use Greenqloud's Storage API with s3cmd and s3fs on Centos 5.6:

Step 1

Log into the Management Console: http://manage.greenqloud.com.

Step 2

Click on "Account information" in the dropdown menu under your username.

Next, download the tar.gz install script pack. This contains install scripts and config files.

Step 3

Unpack and save the folder to your chosen location e.g. "gq_install_scripts". 

Step 4

The install scripts: "s3cmd_install_centos" and "s3fs_install_centos" will install these programs for you and put the config files into the correct locations and backup the default config files.

Step 5

To run these scripts you have to make them executable. Use the command:

chmod +x s3cmd_install_centos s3fs_install_centos

Step 6

To run the scripts you can use the commands: 

./s3cmd_install_centos

and

./s3fs_install_centos

Step 7

After running those scripts the programs should have installed correctly.

Step 8

By using the commands "s3cmd --help"  and "s3fs --help" you can see the possible commands with those programs.

s3cmd usage examples

The command:

s3cmd la

lists all your buckets with their items and

s3cmd ls

lists all your buckets

The command:

s3cmd mb s3://testbucket

creates the bucket "testbucket" and you can insert files into the bucket with

s3cmd put somefile s3://testbucket

(this command inserts the file "somefile" into the bucket "testbucket" ).

s3cmd sync usage examples

This is an extremely useful command which can sync an entire directory to a bucket in Greenqloud's Storage.

 

In a nutshell it works like this:

s3cmd sync      FROM     TO 

To sync a directory /home/user/data/ on your computer to a bucket named testbucket on Greenqloud's storage use:

s3cmd sync   /home/user/data/   s3://testbucket

To sync a bucket named testbucket on GreenQloud's storage to a directory /home/user/data/ on your computer use:

s3cmd sync  s3://testbucket     /home/user/data/

 

s3cmd optional parameters

--dry-run

s3cmd sync  --dry-run /home/user/data/   s3://testbucket

When using --dry-run, s3cmd does'nt upload or download any files, only shows you what will be done if you skip the parameter.

 

--delete-removed

s3cmd sync  --delete-removed /home/user/data/   s3://testbucket

When using --delete-removed files in the target directory which are not in the source directory are deleted

 

 --skip-existing

s3cmd sync  --skip-existing /home/user/data/   s3://testbucket

When using --skip-existing s3cmd only checks if filenames are the same and doesn't check whether the files themselves have changed.

 

Of course you can use all these parameters combined:

s3cmd sync --dry-run --skip-existing --delete-removed   /home/user/data/   s3://testbucket

 

And you can sync the other way around as well with these parameters:

s3cmd sync --dry-run --skip-existing --delete-removed   s3://testbucket    /home/user/data/

 

For further reading on s3cmd sync see:

http://s3tools.org/s3cmd-sync

s3fs usage examples

If you have a bucket in Greenqloud's Storage (here you can see how to create a bucket from the management console: Creating a Bucket ) you can mount it to a directory on your hard drive with s3fs.

To do that you have to create the directory first, e.g. /mount/gq/

If the name of your bucket is "testbucket"  then be root and run the command:

s3fs -o url=http://s.greenqloud.com testbucket /mount/gq/

and then you should be able to access your bucket from the folder /mount/gq/ on your hard drive.

s3fs optional parameters

You can add optional parameters to the s3fs command like:

-o passwd_file

s3fs -o passwd_file=/location/of/the/file/_passwd-s3fs -o url=http://s.greenqloud.com testbucket /mount/gq/

With this paramter you can use any password file you want to, not just the default one which is either in /etc/passwd-s3fs or /home/yourusername/.passwd-s3fs

-o use_cache

s3fs -o use_cache -o url=http://s.greenqloud.com testbucket /mount/gq/

With this parameter all files are cached onto your hard drive making it faster to load them

Warning! Be sure to have enough disk space if you use this command!

-o allow_other

s3fs -o allow_other -o url=http://s.greenqloud.com testbucket /mount/gq/

With this paramter other users can read the mounted drive.

The following parameters are used like the previous ones:

-o default_acl=private 

This is the default setting. Owner gets FULL_CONTROL. No one else has any access rights.

-o default_acl=public_read

Owner gets FULL_CONTROL and the anonymous principal is granted READ access. If this policy is used on an object, it can be read from a browser with no authentication.

-o default_acl=public_read_write

Owner gets FULL_CONTROL, the anonymous principal is granted READ and WRITE access. This is a useful policy to apply to a bucket, if you intend for any anonymous user to PUT objects into the bucket.

-o default_acl=authenticated_read

Owner gets FULL_CONTROL, and any principal authenticated as a registered GreenQloud Storage user is granted READ access.

Automatic mount on boot:

To automatically mount a Storage bucket from GreenQloud on startup, be root and open the file /etc/fstab with

vim /etc/fstab 

Then add this line to the bottom of the file (remember to change "testbucket" to the name of your bucket and "/mount/gq" to your chosen location) :

s3fs#testbucket /mount/gq fuse allow_other,url=http://s.greenqloud.com 0 0

For further reading see:

http://code.google.com/p/s3fs/wiki/FuseOverAmazon