Single Blog

How to Mount S3 Bucket on Linux (AWS EC2 Instance) Use Case:

Use Case:

AWS S3 is an awesome resource for cloud object storage and the consumption of S3 varies from customer, very common use cases were:

  • Backup and Storage – Provide data backup and storage services for others.
  • Application Hosting – Provide services that deploy, install, and manage web applications.
  • Media Hosting – Build a redundant, scalable, and highly available infrastructure that hosts the video, photo, or music uploads and downloads.
  • Software Delivery – Host your software applications that customers can download.

Here I will explain how we implemented S3 in one of our customer.

Customer Requirement:

Customers have an application index logs in the AWS EC2 application cluster which have 4 instances as part of the cluster, each application server logs need to stored centrally and need to accessible (Read-Only) frequently from all servers.
We can consider the NFS sort of solution, even now we have EFS from Amazon but it costly and even the same data were using for their analytics solution. So we thought to use S3 to satisfy both the requirement.

Solution:

We have mount S3 on all required Linux EC2 instances using S3fs so that all required instances have access to logs at the same time their analytic solution also can read data using s3api.
Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual file-system to the Linux kernel. It also aims to provide a secure method for nonprivileged users to create and mount their own file-system implementations.
S3fs-fuse project is written in python backed by Amazons Simple Storage Service. Amazon offers an open API to build applications on top of this service, which several companies have done, using a variety of interfaces (web, rsync, fuse, etc).

Below are the pre-requisites to install and setup S3fs:

  • EC2 instance with root access or sudo access to install S3fs and mount volume
  • IAM user which have S3 Full access (For Upload/Download)
  • Download latest S3fs package from http://code.google.com/p/s3fs/downloads/list 
  • update your system to the latest using

yum update all (for CentOS)

         apt-get update (for Ubuntu)
  • Install below dependencies before installing S3fs package

For CentOS 

yum install gcc libstdc++–devel gcc–c++ fuse fuse–devel curl–devel libx
ml2–devel openssl–devel mailcap

For Ubuntu

apt-get install build-essential gcc libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support pkg-config libxml++2.6-dev libssl-dev

Follow the below steps to mount your S3 bucket to your Linux Instance:

 Step 1: Download the latest s3fs package and extract:

wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/s3fs/s3fs-1.74.tar.gz

 

tar -xvzf s3fs-1.74.tar.gz

Step 2: Update OS and install dependencies as mentioned above pre-req.

Step 3:  Now change to extracted directory, compile and install s3fs source code.

cd s3fs-1.74

 

./configure –prefix=/usr

 

make

 

make install


Step 4: Use the below command to check where s3fs command is placed in O.S. It will also confirm whether the installation is ok:

which s3fs


Step 5: Get IAM user Access and secret key which have appropriate permissions (e.g. S3 Full access), You can get the same from the AWS IAM console

Step 6: Create a new file in /etc with the name passwd-s3fs and Paste the access key and secret key in the below format and change the permission for the file:

echo “AccessKey:SecretKey” > /etc/passwd-s3fs

 

chmod 640 /etc/passwd-s3fs

Note: Replace AcessKey and SecretKey with original keys.

Step 7: Now create a directory and mount S3bucket in it. Here, Provide your S3 bucket name in place of “your_bucketname”

mkdir /sravancloudarch


s3fs your_bucketname -o use_cache=/tmp -o allow_other -o multireq_max=5 /mys3bucket

Replace your_bucket = S3 bucket name which you want to mount
Replace /sravancloudarchS3 = Directory name which you want to mount

/usr/bin/s3fs sravancloudarch -o use_cache=/tmp -o allow_other -o multireq_max=5 /sravancloudarch

 

You can validate whether it is mounted using below command:
 
[root@ip-172-31-49-68 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        488M   56K  488M   1% /dev
tmpfs           497M     0  497M   0% /dev/shm
/dev/xvda1      7.8G  1.1G  6.6G  15% /
s3fs            256T     0  256T   0% /mys3bucket
s3fs            256T     0  256T   0% /sravancloudarch
[root@ip-172-31-49-68 ~]#


Note: At any given point you can unmount this volume using the below command:

umount /sravancloudarch

 

Now, this volume is Non-Persistent i.e; once you reboot your system this mount point won’t exist, to make it persistent and automatically mount for every reboot we need to add below entries to /etc/rc.local
 
nano /etc/rc.local


Add below Line and save the file:

/usr/bin/s3fs sravancloudarch -o use_cache=/tmp -o allow_other -o multireq_max=5 /sravancloudarch

 

Now you should be able to read and write files to S3 (Considering you have S3 full access).
touch /sravancloudarchS3/sravan
 
[root@ip-172-31-49-68 ~]# ls -lrt /sravancloudarch/
total 1
-rw-r–r– 1 root root 0 Jun  7 10:11 sravan
[root@ip-172-31-49-68 ~]#
 


Now you have successfully mounted your S3 bucket as a Volume in your EC2 instance, any files which you write in /sravancloudarch directory will be replicated to your Amazon S3 bucket.

Hopefully, you enjoyed this post, here are the relevant hands-on video 

<>

Comments (0)

Post a Comment