Amazon S3 分布式存儲的 python 接口實現

Amazon s3 是一種分布式的對象存儲。用鍵值對的方式，來存儲數據。其中，存入的所有數據都是一個對象（object），每一個對象都有一個鍵（key）存在。

具有非常方便的 web 訪問接口，以及權限控制。

 

下面是具體讀、寫、判斷三個接口的具體實現

 

1. 創建一個 bucket

class ImageFeatIO:
    __read_singleton = None
    __write_singleton = None
    __read_count = 0
    __write_count = 0

    def __init__(self):
        pass

    @staticmethod
    def get_write_instance():  #創建一個 bucket 的單例
        if ImageFeatIO.__write_singleton is None:
            paras = get_config('config')
            access_key = paras['access_key']
            secret_key = paras['secret_key']
            write_host = paras['file_write_host']
            conn = boto.connect_s3(
                aws_access_key_id=access_key,
                aws_secret_access_key=secret_key,
                host=write_host, is_secure=False,
                calling_format=boto.s3.connection.OrdinaryCallingFormat()
            )
            bucket_name = paras['bucket_name']
            bucket = conn.get_bucket(bucket_name)
            ImageFeatIO.__write_singleton = bucket
        return ImageFeatIO.__write_singleton

 

2. 寫入文件

寫入文件的時候，可以進行權限控制

官方說明如下：

1. Create a custom ACL that grants specific rights to specific users. At the moment, the users that are specified within grants have to be registered users of Amazon Web Services so this isn’t as useful or as general as it could be.
2. Use a “canned” access control policy. There are four canned policies defined:
1. private: Owner gets FULL_CONTROL. No one else has any access rights.
2. public-read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.
3. public-read-write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.
4. authenticated-read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access

def write_image_feature_to_file(id, imageFeaturestring):
    bucket = ImageFeatIO.get_write_instance()
    k = Key(bucket)
    k.key = id
    res = k.set_contents_from_string(imageFeaturestring)
    k.set_acl('authenticated-read')
    return res

 

3.讀取（下載）文件

s3 提供了一種非常方便的 web 服務接口，可以從任何地方以 http 協議獲取數據

def read_image_feature_from_file(id):
    url_head = ImageFeatIO.get_read_instance()
    url = url_head + '/' + id  #url= http://img.3weijia.com/bucket_name/key
    request = urllib2.Request(url=url)
    response = urllib2.urlopen(request, timeout=20)
    res_unicode = unicode(response.read())
    res = res_unicode.encode('unicode-escape').decode('string_escape')
    return res

 

4. 判斷鍵值是否存在

def if_image_feature_exist(id):
    bucket = ImageFeatIO.get_write_instance()
    key = bucket.get_key(id)

    return key is not None

 

參考：

http://boto.readthedocs.io/en/latest/s3_tut.html

https://www.cnblogs.com/web424/p/6840207.html