using:

/usr/bin/s3fs mybucket -o accessKeyId=aaa -o secretAccessKey=bbb /mnt

If you don't like specifying your secretAccessKey on the command line then you
can create a file "/etc/passwd-s3fs" with a line containing a
accessKeyId:secretAccessKey pair. Then the command line becomes simply:

You can have more than one set of credentials (i.e., credentials for more than
one amazon s3 account) in /etc/passwd-s3fs in which case you'll have to specify
-o accessKeyId=aaa on the command line.

s3fs has a caching mechanism: You can enable local file caching to minimize downloads, e.g., :
/usr/bin/s3fs mybucket /mnt -ouse_cache=/tmp


Updated Feb 04, 2010 by rrizun
Labels:	Featured
FuseOverAmazon  
FUSE-based file system backed by Amazon S3

For a commercially supported version with additional features see http://www.subcloud.com

What's New

r191 merged in tractis eu bucket changes http://github.com/tractis/s3fs-fork
r188 includes eucalyptus patch http://open.eucalyptus.com/wiki/s3fs
r177 fixed stale curl handle issue; fixed 100% cpu issue
r166 case insensitive mime type lookup
r152 (May 8, 2008)
uses x-amz-copy-source; "correct" content-type lookup via /etc/mime.types; symlinks!
Overview

s3fs is a fuse filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files). Maximum file size=5G.

s3fs is stable and is being used in number of production environments, e.g., rsync backup to s3.

To use it:

get an amazon s3 account!
download the source, compile it (I've used fc5/ppc, f7/i386, f9/x86, f9/x64 and Mac OS X 10.4) and slap the binary in, say, /usr/bin/s3fs
you'll need at least fuse-2.6
for fedora probably need to do: yum install fuse-devel
for ubuntu probably need to do something like: sudo apt get fuse-libs (I think?!?)
do this:
/usr/bin/s3fs mybucket -o accessKeyId=aaa -o secretAccessKey=bbb /mnt
That's it! the contents of your amazon bucket "mybucket" should now be accessible read/write in /mnt

If you don't like specifying your secretAccessKey on the command line then you can create a file "/etc/passwd-s3fs" with a line containing a accessKeyId:secretAccessKey pair. Then the command line becomes simply:

/usr/bin/s3fs mybucket /mnt
You can have more than one set of credentials (i.e., credentials for more than one amazon s3 account) in /etc/passwd-s3fs in which case you'll have to specify -o accessKeyId=aaa on the command line.

s3fs supports mode (e.g., chmod), mtime (e.g, touch) and uid/gid (chown). s3fs stores the values in x-amz-meta custom meta headers, and as such does "brute-force" re-uploads of s3 objects if/when mode and/or mtime changes. and uses x-amz-copy-source to efficiently change them.

s3fs has a caching mechanism: You can enable local file caching to minimize downloads, e.g., :

/usr/bin/s3fs mybucket /mnt -ouse_cache=/tmp
Hosting a cvsroot on s3 works! Although you probably don't really want to do it in practice. E.g., cvs -d /s3/cvsroot init. Incredibly, mysqld also works, although I doube you really wanna do that in practice! =)

Using rsync with an s3 volume as the destination doesn't quite work because of timestamp issues. s3fs does not (yet) support changing timestamps on files. I mean, it will work, as in it will copy files, but, the timestamps will just be current timestamps (rsync will complain about not being able to set timestamps but will continue).

s3fs works with rsync! (as of svn 43) Due to the way FUSE works and s3fs' "brute-force" support of mode (chmod) and mtime (touch), upon first sync, files are downloaded/uploaded more than once (because rsync does (a) chmod (b) touch and (c) rename), however, subsequent rsyncs are pretty much as fast as can be. If that's too much downloading/downloading for ya then try using the "use_cache" option to enable the local file cache... it will definitely minimize the number of downloads. as of r152 s3fs uses x-amz-copy-source for efficient update of mode, mtime and uid/gid.

s3fs will retry s3 transactions on certain error conditions. The default retry count is 2, i.e., s3fs will make 2 retries per s3 transaction (for a total of 3 attempts: 1st attempt + 2 retries) before giving up. You can set the retry count by using the "retries" option, e.g., "-oretries=2".

Options

accessKeyId
secretAccessKey
default_acl (default="private")
the default canned acl to apply to all written s3 objects, e.g., "public-read"
see http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?RESTAccessPolicy.html "Canned Access Policies" for the full list of canned acls
any created files will have this canned acl
any updated files will also have this canned acl applied!
prefix (default="") (coming soon!)
a prefix to append to all s3 objects
retries (default="2")
number of times to retry a failed s3 transaction
use_cache (default="" which means disabled)
local folder to use for local file cache
connect_timeout (default="2" seconds)
time to wait for connection before giving up
readwrite_timeout (default="10" seconds)
time to wait between read/write activity before giving up
url (default="http://s3.amazonaws.com")
sets the url to use to access amazon s3, e.g., if you want to use https then set url=https://s3.amazonaws.com

