AWS Cached Volume Storage Gateway Walk Through

AWS Cached Volume Storage Gateway

What Is AWS Storage Gateway?

Volume Gateway — A volume gateway provides cloud-backed storage volumes that you can mount as Internet Small Computer System Interface (iSCSI) devices from your on-premises application servers. The gateway supports the following volume configurations:

  • Cached volumes — You store your data in Amazon Simple Storage Service (Amazon S3) and retain a copy of frequently accessed data subsets locally. Cached volumes offer a substantial cost savings on primary storage and minimize the need to scale your storage on-premises. You also retain low-latency access to your frequently accessed data.

Steps for this walk through:
1. Beginning provisioning of Cached Volumes Storage Gateway.
2. Download VMWare Esxi Gateway appliance from AWS.
3. Install Gateway appliance locally. Your PC or on-premises server. For this walk through, I will use VMWare Workstation on my PC.
4. Build up and configure Gateway appliance locally.
5. Connect Gateway appliance to AWS Storage Gateway.
6. Create Volume from AWS Storage Gateway Console.
7. Connect your PC to local Gateway appliance through iSCSI connector, initiate the connected volume.
8. Verify everything is working correctly and start uploading files to S3 through Gateway appliance!

Extension usage:
1. Make a snapshot of the volume with data uploaded from AWS Storage Gateway console.
2. Create a EBS volume out from the snapshot.
3. Attach the volume to an EC2 and mount the volume.
4. Verify if the uploaded data is seen correctly!

Basic concept of Cached Volumes Storage Gateway(C.V.S.G) is that, on the gateway appliance, you need to configure two disks, one for cache storage and one for upload buffer. C.V.S.G stores all your data in AWS S3 while keeping a copy of recent accessed data locally on CACHE STORAGE for low latency fast access. Where data is uploaded from and download to UPLOAD BUFFER when your application writes and retrieves data from AWS S3. For more details, refer Cached Volumes Architecture.

Only main/key actions are shown below. Start by creating C.V.S.G.

AWS Cached Volume Storage Gateway

Options for building your local gateway appliance. For this walk through, I chose VMware ESXi.

AWS Cached Volume Storage Gateway

Once the OVF template is download, import it to VMware. My case, VMware Workstation player on my local PC.

AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway

Once imported, create two disks for the gateway appliance as cached storage and upload buffer. Make sure your gateway appliance has 4 core and at least 16 GB ram.

AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway

The recommended size for both disks is150GB but I configured 20 GB for demo.

AWS Cached Volume Storage Gateway

Repeat the steps twice to create two extra disks. Power on the VM once done.

AWS Cached Volume Storage Gateway

At the login screen, the account name is sguser and password is sgpassword. Once login, check your network Connectivity and sycn your system time. Copy down your IP address as you will need it later.

AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway

Once done with Gateway appliance, contine from AWS Storage Gateway Console and input your Gateway appliance IP you copied earlier.

AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway

At this moment, AWS Storage Gateway and Gateway appliance is done setup and connected. Next, we create a storage volume for the data to be async from on-premises to S3.

AWS Cached Volume Storage Gateway

Set a name and size for your storage volume. For my demo, I created a 10 GB storage volume. Skip CHAP authentication for now, we will do it later.

AWS Cached Volume Storage Gateway

Finish view of storage volume.

AWS Cached Volume Storage Gateway

Now back to local PC, we need to configure PC to connect to Gateway appliance through iSCSCI connector. For windows, from search, key in is and the iSCSI connector program will popup.

For English version, please refer to Connecting to Your Volumes to a Windows Client.
For Linux, please refer to Connecting Your Volumes or VTL Devices to a Linux Client.

AWS Cached Volume Storage Gateway

Follow the steps to setup connector. First discover, at step 3, fill in the Gateway appliance IP.

AWS Cached Volume Storage Gateway

Now connect, the status of iSCSI connector is connected on the right image.

AWS Cached Volume Storage Gateway

Next, we need to initiate the disk we connected to through iSCSI so we can start using it.

AWS Cached Volume Storage Gateway

For disk size smaller then 2TB, choose MBR, bigger than 2TB, choose GPT.

AWS Cached Volume Storage Gateway

Select the 10 GB storage volume created earlier from AWS Storage Gateway console and initiate it, click NEXT till the end.

AWS Cached Volume Storage Gateway

Once done, we test it by putting some images to the storage volume.

AWS Cached Volume Storage Gateway

Now check on C.V.S.G on AWS console and will see that 33.9 MB of data is put into upload buffer.

AWS Cached Volume Storage Gateway

A while later, data will be transferred to storage volume from upload buffer.

AWS Cached Volume Storage Gateway

That’s it! We have successfully setup C.V.S.G between on-premises and AWS cloud.

Next, we try out the snapshot function by creating a snapshot of the storage volume.

AWS Cached Volume Storage Gateway

Quickly check on the snapshot page and see that a snapshot is creating. Once it is done, create a EBS volume out from it from Actions.

AWS Cached Volume Storage Gateway

Attach the created EBS volume to an EC2.

AWS Cached Volume Storage Gateway
AWS Cached Volume Storage Gateway

Lastly, login to the EC2, mount the EBS volume and verify the content!

AWS Cached Volume Storage Gateway

For better security, CHAP authentication is recommended. Right now the connection is not CHAP authenticated.

AWS Cached Volume Storage Gateway

Start by getting your local iSCSI initiate’s name. At step 4, set a password, for example, sec2ndpassword.

AWS Cached Volume Storage Gateway

Now we set up CHAP authentication on AWS console.

AWS Cached Volume Storage Gateway

For initiator name, fill in the name you just copied from step 2 and sec2ndpassword to Target secret. Set a password to initiator secret, for example, first1stpassword. Once done, you will see that CHAP auth has changed to Yes.

AWS Cached Volume Storage Gateway

Now back to your iSCSI connector and connect again with CHAP Authentication. When connecting, click on advance(A) show in left image, key in first1stpassword in the TargetSecret(S) in right image.

AWS Cached Volume Storage Gateway

Once connected, verify that the authentication is on!

AWS Cached Volume Storage Gateway

And that’s it! Now you are transferring data with higher security!

Reference:
Hands-On Lab with AWS Storage Gateway part 1 - Gateway-Cached

AWS Certified SA, SysOps & Developer Associate, Alibaba Cloud certified SA. Focusing on Azure, Prometheus w/ Grafana, ELK and K8S now.