Thursday 9 February 2012

Mount Amazon S3 storage as local filesystem on Ubuntu server

Amazon S3 is a high reliability online storage. We have recently started using Amazon S3 in our company as a backup storage for our projects and as a storage for static sites content. Next step-by-step tutorial describes how to mount S3 bucket as a local filesystem using FUSE-based file system s3fs.

1. Install neccessary packages
$ sudo apt-get update
$ sudo apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support
2. Next step is to download archive with latest version of s3fs
$ wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
$ tar xzvf s3fs-1.61.tar.gz
3. Compile sources
$ cd s3fs-1.61/
$ sudo ./configure
$ sudo make
$ sudo make install
4. Allow other users have access to s3 bucket that will be mounted

Edit file /etc/fuse.conf with any text editor, I used vim for this:
$ sudo vim /etc/fuse.conf
And uncomment the following line in the conf file:
...
#user_allow_other

5. Add pare of you Amazon account key id and account access key to /etc/passwd-s3fs file

Edit file /etc/passwd-s3fs and add here string: AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY
Where: AWS_ACCESS_KEY_ID -- your amazon account key id
AWS_SECRET_ACCESS_KEY -- your amazon account access key

6. Set permissions to file
$ sudo chmod 0600 /etc/passwd-s3fs    
7. Mount s3 bucket to local filesystem
$ sudo s3fs your_backet_name -o use_cache=/tmp -o allow_other /mnt/s3storage
That's all and now S3 bucket can be used as local filesystem.

33 comments:

  1. Hi,

    S3fs works well for most use cases, but there is one thing you should be aware of, it uses their own format for folders emulation, so, folders created by s3fs are not recognized by another s3 clients including AWS Management Console.

    Here is the discussion related to this issue:
    https://forums.aws.amazon.com/thread.jspa?threadID=58468

    If you are on Windows, you can use TntDrive (it's not free) to mount Amazon S3 Bucket as a network or removable drive: http://tntdrive.com

    ReplyDelete
    Replies
    1. Hm, very interesting. Will keep it in mind. Thanks for your comment.

      Delete
    2. I got an error when mouting

      unable to access MOUNTPOINT allow_other: No such file or directory

      Delete
    3. For reference, there exists a fork of the s3fs project (as of version 1.59) at:

      https://github.com/tongwang/s3fs-c

      That aims to be compatible with s3 clients such as the AWS Management Console.

      I was not able to try it because I have an oldish OS version that it does not support.

      Philippe Paravicini
      januxmedia.com

      Delete
  2. I think their is a typo last line should be:
    sudo s3fs your_bucket_name -o use_cache=/tmp -o allow_other /mnt/s3storage

    ReplyDelete
    Replies
    1. Yes, you'r right. Will fix this in the post. Thanks.

      Delete
  3. Thanks a bunch for sharing this with all folks you actually know what you are talking about! Bookmarked. Please additionally seek advice from my website =). We could have a hyperlink trade arrangement among us
    Download Free Movie Online, Surely Good Sources

    ReplyDelete
  4. Thanks for the sharing of such information. we will pass it on to our readers. This is a great reading. Thanking you.
    web design company in delhi

    ReplyDelete
  5. Does anyone have any experience with this when you have multiple EC2 instances running on a load balancer? Trying to figure out if the buckets should be mounted on ALL servers, or just the primary server...

    ReplyDelete
    Replies
    1. Hi Robert. It is necessary to mount bucket to all your EC2 instances. Load balancer just passes traffic to one of the EC2 instances via http, and doesn't know anything about the filesystem or what storages are mounted to any of the servers under this load balancer.

      Btw., S3 bucket is good only storing static content. We tried to use it as webroot directory shared between instances for easy deployment and this caused a lot of problems. But for static content it's a really good thing.

      Delete
  6. S3fs works well for most utilization cases, however there is one thing you ought to be mindful of, it uses their organization for organizers imitating, along these lines, envelopes made by s3fs are not distinguished by an alternate s3 customers including AWS Management Console.

    Best Website Design//Mobile Apps N Webs Development

    ReplyDelete
  7. Much obliged for imparting this to all people you really recognize what you are discussing! Bookmarked. It would be ideal if you moreover look for guidance from my site =). We could have a hyperlink exchange game plan around us
    Best Web Design

    ReplyDelete
  8. This really nice and innovative post on php web development.

    Thanks
    rickybroad

    ReplyDelete
  9. If you want to make sure that the sf3s mount is available in the event that the computer reboots, you can put the mount command under /etc/rc.local.

    Philippe Paravicini
    januxmedia.com

    ReplyDelete
  10. Hi, you used to make awesome content, but the last few content has been kind of boring… I miss your remarkable content. Previous several components are just a bit out of track!

    website development companies Bangalore | website designing Bangalore

    ReplyDelete
  11. Thank you for your post, I look for such article along time, today i find it finally. this post give me lots of advise it is very useful for me
    Signature:
    i like play games happy wheels online and play happy wheels 2 games and zombie tsunami game , retrica camera , retrica , happy wheels , agario

    ReplyDelete
  12. One of the nice information you have shared thanks for sharing this information keep share more information like that...
    Bangalore Web Development Companies | Website Development Company in Bangalore

    ReplyDelete
  13. Web Developing field becoming huge growth because of increase the demand of websites.

    Web Design Company in Delhi | Website Designing Company in Delhi

    ReplyDelete
  14. تعتبر العاب تلبيس من اشهر الانواع في هذا المجال وهي بدورها تتضمن عدة اصناف جميلة ويعشقها الكتير وخاصة البنات منها العاب تلبيس ومكياج التي تمزج بين التلبيس وكذلك الميك اب في آن واحد هذا الامر الدي يزيد من جمالها وتجعل كل من يلعبها يستمتع بذلك زد على ذلك العاب تلبيس باربي التي تعرف شعبية كبيرة لانها شخصية مشهورة ويعرفها الصغير والكبير ولهم ذكريات جميلة معها لانها اشتهرت في عالم الكارتون والان اصبح الامر كذلك في مجال الالعاب وغير هذا هناك كذلك نوع آخر مميز ايضا وهو العاب تلبيس عرائس فالجميع يحلم ان يقوم بتلبيسهما لانها تذكرهم بهذه المناسبة الجميلة
    al3ab-banat01

    ReplyDelete
  15. What if the command for 'Set permissions to file' which is $ sudo chmod 0600 /etc/passwd-s3fs not working debian. Because of this error mounting amazon s3 storage as local is not being done.

    Web development company in Indore

    ReplyDelete
  16. As an Internet marketing strategy, SEO considers how search engines work, the computer programmed algorithms which dictate search engine behavior, what people search for Seo in bangalore, the actual search terms or keywords typed into search engines, and which search engines are preferred by their targeted audience. Optimizing a website may involve editing its content, adding content, doing HTML, and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines.

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Hi! Thank you for the share this information. This is very useful information for online blog review readers. Keep it up such a nice posting like this. We are most leading IT & Software company in India

    ReplyDelete
  19. Thank you for writing this informative post. Looking forward to read more.
    Best Web Development Company in India

    ReplyDelete
  20. Thanks for providing such a valuable knowledge on Web Development

    ReplyDelete
  21. Thank you for sharing such useful information. I really enjoyed while reading your article and it is good to know the latest updates. Do post more. And also get to know about the best
    Web Development Company in India Who offers best and affordable Website Development Services

    ReplyDelete
  22. Thanks for sharing such a good article. It was well explained and valuable. Nice article on Web Development .

    ReplyDelete
  23. Nice blog and very informative....

    Looking affordable web development services in USA? Visit once - https://meerakics.com/ or Contact us - +91 7018300334

    ReplyDelete
  24. Nice Blog. Webcloud Technologies also provides best website designing and development service at best price in india.

    web development company in delhi

    ReplyDelete