Cloud storage using StowUtils package in Golang

The intention of this article is to describe how to store files in cloud storage using the Golang stowutils package.

 

How does it work?

 

Implementations are referred to by a kind of string (like s3, google) and an object which provides specific configuration values associated with the implementation (like the S3 region, the credentials, the project ID in case of Google Cloud, etc.)

 

It uses stow package.

 

Stow provides implementations for cloud storage services. All providers almost have a similar approach when it comes to storing content files (or items) and are stored in buckets (or containers).

 

Object storage services are the concept around which the idea of stow is structured. It is made up of three main objects:

 

  • Location – The Container objects are stored here.
  • Container – Group of Item objects.
  • Item – an individual file.

 


 

location1 (e.g. s3)

container1

item1.1

item1.2

container2

item2.1

item2.2

location2 (e.g. google)

container1

item1.1

item1.2

container2

item2.1

item2.2

 

 

It should be noted that:

 

  • Location is like s3 or Google, and each location may have many containers (or storage buckets).
  • A container contains many items ( or files).
  • Containers do not contain other containers.
  • Items must belong to a container.

 

Pre-requisite:

 

Import Stowutils and implementations:

 

First, you import stowutils and any implementations you wish to support:

 


 

import (

utils "github.com/agiratech/stowutils"

)

 

 

Features of the Stowutils package:

  1. Get Location
  2. Acquire list of containers
  3. Get Container
  4. Acquire list of Items
  5. Get Item
  6. Upload Item
  7. Read Item
  8. Download Item
  9. Delete Item

 

Get Location

 

In order to get a location, you need to know the kind of string and a utils.Config object that contains the required configuration information (such as AccessKey, secretKey, JSON Credentials, projectID, etc). Configuration implementation is specific, and so, you should consult each implementation to see what fields are required for particular kind.

 

Example: For google cloud storage

 


 

g_location, err := utils.GetLocation("google",&utils.Config{

GoogleConfigJSONFilePath:"json-credentials.json",

GoogleProjectID:"sample-project"})

if err != nil {

return err

}

defer g_location.Close()

// use g_location

 

 

Example: For s3 cloud storage

 


s_location, err := utils.GetLocation("s3", &utils.Config{

S3AccessKeyID:"1452468515",

S3SecretKey:"a1b2c3d4",

S3Region:"us-west-2"})

if err != nil {

return err

}

defer s_location.Close()

// use s_location

 

 

Get list of containers

 

You can get Container list using the utils.ContainerList function:

 


 

func ContainerList(location stow.Location, prefix string, pageSize int) ([]stow.Container, error)

 

 

Here, prefix is a prefix name of containers.

 

Example:

 


 

g_containers, err := utils.ContainerList(g_location, "", 100)

if err != nil {

return err

}

for _,c := range g_containers {

// use container c

}

 

 

Get Container

 

You can get a required container using the utils.GetContainer function:

 


 

func GetContainer(location stow.Location, container_name string, pageSize int)
(stow.Container, error) {

 

 

Here, container_name is a required container name (and hence the container_name should be available)

 

Example:

 


 

g_container, err := utils.GetContainer(g_location, "sample-container", 100)

if err != nil {

return err

}

// use container

 

 

Get a list of Items

 

You can get item list using utils.ItemList function:

 


 

func ItemList(container stow.Container, prefix string, pageSize int) ([]stow.Item, error) {

 

 

Here, prefix is the prefix name of items.

 

Example:

 


 

g_items, err := utils.ItemList(g_container, "", 100)

if err != nil {

return err

}

for _,i := range g_items {

// use item i

}

 

 

Get Item

 

You can get a particular item using utils.GetItem function:

 


 

func GetItem(container stow.Container, item_name string, pageSize int) (stow.Item, error) {

 

 

The item_name mentioned here is a required item name (and hence the item_name should be available)

 

Example:

 


 

g_item, err := utils.GetItem(g_container, "img024.jpg", 100)

if err != nil {

return err

}

// use item

 

 

Upload Item

 

You can upload an item(file) using utils.UploadItem function:

 


 

func UploadItem(container stow.Container, file_path string, file_name string, metadata
map[string]interface{}) (stow.Item, error) {

 

 

Into whichever container you want to upload a file, pass that container as parameter. Here, file_path is an uploading file path and file_name is the name of the file, which should be unique. Otherwise, it will override an existing file.

 

Example:

 


 

item, err := utils.UploadItem(g_container, "/home/images/image001.jpg", "image001.jpg", nil)

if err != nil {

return err

}

// check the item

 

 

Read Item

 

You can read the item information using utils.ReadItem function:

 


 

func ReadItem(g_item stow.Item) (io.ReadCloser, error) {

 

 

Here, g_item is a received item using utils.GetItem

 

Example:

 


 

file, err = utils.ReadItem(item)

if err != nil {

return err

}

// use file

 

 

Download Item

 

Also, you can get the URL of the file using utils.DownloadItem function:

 


 

func DownloadItem(item stow.Item) *url.URL {

 

 

Example:

 


 

url = utils.DownloadItem(item)

// use URL

 

 

Delete Item

 

You can delete an item from a container using utils.DeleteItem function as mentioned below:

 


 

func DeleteItem(container stow.Container, item_name string) error {

 

 

Pass that container and file_name into the function as a params (the file_name should be available), which contains the file that you want to delete.

 

Example:

 


 

err = utils.DeleteItem(g_container, item.Name())

if err != nil {

return err

}

 

Conclusion

The above mentioned are all the features of the Stowutils package, and this article covers the basics of how it can be done, and what are the possibilities with it in Golang. Thus, the Stowutils package can be used to achieve the storing of files in the Cloud Storage. Try it out yourself for a better understanding.