java實現簡單的oss存儲

oss

工作中需要用到文件上傳，之前使用的是本地文件系統存儲方式，后來重構為支持多個存儲源的方式，目前支持三種方式：local、seaweedfs、minio

存儲介質

seaweedfs

seaweedfs是一款go語言開發的輕量級、高性能的存儲服務器。
https://github.com/chrislusf/seaweedfs

# 啟動 master
docker run -d \
-p 9333:9333 \
-p 19333:19333 \
-v /home/mining/report-cloud/docker/seaweedfs/master/data:/data \
--name seaweedfs-master \
chrislusf/seaweedfs:1.53 \
master -ip=master

# 啟動 volume
docker run -d \
-p 8080:8080 \
-p 18080:18080 \
-v /home/mining/report-cloud/docker/seaweedfs/volume01/data:/data \
--name seaweedfs-volume \
--link seaweedfs-master:master \
chrislusf/seaweedfs:1.53 \
volume -max=5 -mserver="master:9333" -port=8080

訪問web地址：http://localhost:9333/
簡單使用：

# 申請空間
curl http://localhost:9333/dir/assign
#
{"count":1,"fid":"4,017f52110b","url":"127.0.0.1:8080","publicUrl":"localhost:80
80"}
# 推送文件，4,017f52110b表示第4個volume，017f52110b表示文件的唯一標識
curl -F file=@/home/seaweedfs/balance.png http://127.0.0.1:8080/4,017f52110b

minio

MinIO是與Amazon S3 API兼容的高性能對象存儲服務器，提供了人性化的管理頁面
https://github.com/minio/minio

docker run --name report-minio \
-p 19000:9000 \
-e "MINIO_ACCESS_KEY=qweasdzxc" \
-e "MINIO_SECRET_KEY=1234567890" \
-v /home/mining/report-cloud/docker/minio:/data \
-d minio/minio:latest server /data

# web 管理頁面
http://localhost:19000/minio/login
# 創建一個 bucket 為 report
# 通過URL直接訪問文件需要設置權限，參考下面博客
# https://blog.csdn.net/iKaChu/article/details/105809957
# 訪問格式為
http://localhost:19000/bucket名/對象名

Java代碼

maven依賴

<!-- oss 存儲的依賴 -->
<!-- seaweedfs 依賴 -->
<dependency>
  <groupId>org.lokra.seaweedfs</groupId>
  <artifactId>seaweedfs-client</artifactId>
  <version>0.7.3.RELEASE</version>
</dependency>

<!-- minio 依賴 -->
<dependency>
  <groupId>io.minio</groupId>
  <artifactId>minio</artifactId>
  <version>3.0.10</version>
</dependency>

代碼

配置類

FileTag 枚舉類

用來標記使用哪種存儲介質

/**
 * 文件上傳的tag
 */
public enum FileTag {
    TYPE_LOCAL,
    TYPE_MINIO ,
    TYPE_SEAWEEDFS;
}

FileUploadConfigLocal

本地存儲配置類

public class FileUploadConfigLocal {
    private String parentPath;
    // 省略 get set
}

FileUploadConfigSeaweedfs

seaweedfs存儲配置類

public class FileUploadConfigSeaweedfs {
    private String host;
    private int port = 9333;
    private String publicUrl;
    private int connectTimeout = 60; // 60 s
    // 省略 get set
}

FileUploadConfigMinio

minio存儲配置類

public class FileUploadConfigMinio {
    private String domain; // url 前綴
    private String accesKey;
    private String secretKey;
    private String bucket; // 必須事先創建好
    // 省略 get set
}

FileUploadConfig

spring application.yaml 配置類

import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties( prefix = "oss-config")
public class FileUploadConfig {
    FileTag tag;
    FileUploadConfigLocal local;
    FileUploadConfigSeaweedfs seaweedfs;
    FileUploadConfigMinio minio;
    // 省略 get set
}

配置文件

application.yaml

# 對象存儲配置
oss-config:
  tag: TYPE_LOCAL # 使用哪種存儲介質
  local:
    parent-path: D:\dev\2019-06-19\upload
  minio:
    domain: http://localhost:19000
    bucket: report
    acces-key: qweasdzxc
    secret-key: 1234567890
  seaweedfs:
    host: localhost
    port: 9333
    publicUrl: http://localhost:8080
    connectTimeout: 3000 # 5min

FileUploadInfoVO文件上傳類

public class FileUploadInfoVO {

    private String fileName; // 文件名
    private String suffix; // 文件后綴
    private String contentType; // 文件類型
    private Long size; // 文件大小
    private String fid; // 文件唯一ID，同時也是保存在服務器上的文件名
    private boolean delete; // 是否刪除，默認false
    private String description;
    private String url; // 完整的url
    private String tag; // 標簽，比如
    private String uploadTime;
    private String deleteTime;
    // 省略 get set
}

業務邏輯類

FileStorageService 文件存儲接口

/**
 * 文件存儲的接口
 */
public interface FileStorageService {
    FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception ;
    FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception ;
    FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception ;
}

本地存儲實現

@Service("FileStorageServiceLocal")
public class FileStorageServiceLocal implements FileStorageService {

    @Autowired
    FileUploadConfig fileUploadConfig;

    @PostConstruct
    public void init() {
        FileUtils.checkAndCreateDir(fileUploadConfig.getLocal().getParentPath());
    }

    @Override
    public FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        String filePath = fileUploadConfig.getLocal().getParentPath() + File.separator + fileUploadInfoVO.getFid();
        FileUtils.copyFileFromInputStream(in, filePath);
        fileUploadInfoVO.setUrl(filePath+"."+fileUploadInfoVO.getSuffix());
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        String filePath = fileUploadConfig.getLocal().getParentPath() + File.separator + fileUploadInfoVO.getFid();
        FileUtils.copyFileFromBytes(bytes, filePath);
        fileUploadInfoVO.setUrl(filePath);
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception {
        File file = new File(fileUploadInfoVO.getUrl());
        if(file.exists()) {
            file.delete();
        }
        return fileUploadInfoVO;
    }
}

Seaweedfs存儲實現

@Service("FileStorageServiceSeaweedfs")
public class FileStorageServiceSeaweedfs implements FileStorageService {

    @Autowired
    FileUploadConfig fileUploadConfig;

    FileTemplate fileTemplate;

    @PostConstruct
    public void init() {
        FileUploadConfigSeaweedfs seaweedfs = fileUploadConfig.getSeaweedfs();
        FileSource fileSource = new FileSource();
        fileSource.setHost(seaweedfs.getHost());
        fileSource.setPort(seaweedfs.getPort());
        fileSource.setConnectionTimeout(seaweedfs.getConnectTimeout());//5min
        try {
            fileSource.startup();
        } catch (IOException e) {
            throw new RuntimeException("創建seaweedfs連接失敗，原因是：" + e.getMessage());
        }
        fileTemplate = new FileTemplate(fileSource.getConnection(), seaweedfs.getPublicUrl());
    }

    @Override
    public FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        FileHandleStatus fileHandleStatus = fileTemplate.saveFileByStream(fileUploadInfoVO.getFileName(), in, ContentType.create(fileUploadInfoVO.getContentType(), "utf-8"));
        fileUploadInfoVO.setFid(fileHandleStatus.getFileId());
        fileUploadInfoVO.setUrl(String.format("%s/%s", fileUploadConfig.getSeaweedfs().getPublicUrl(), fileHandleStatus.getFileId()));
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        return upload(new ByteArrayInputStream(bytes), fileUploadInfoVO);
    }

    @Override
    public FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception {
        fileTemplate.deleteFile(fileUploadInfoVO.getFid());
        return fileUploadInfoVO;
    }
}

Minio存儲實現

@Service("FileStorageServiceMinio")
public class FileStorageServiceMinio implements FileStorageService {

    @Autowired
    FileUploadConfig fileUploadConfig;

    @Override
    public FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        FileUploadConfigMinio minio = fileUploadConfig.getMinio();
        String domain = minio.getDomain();
        String bucket = minio.getBucket();
        String fileName = fileUploadInfoVO.getFileName();
        MinioClient minioClient = new MinioClient(domain, minio.getAccesKey(), minio.getSecretKey());
        minioClient.putObject("report", fileName, in, fileUploadInfoVO.getSize(), fileUploadInfoVO.getContentType());
        fileUploadInfoVO.setUrl(String.format("%s/%s/%s", domain, bucket, fileName));
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        return upload(new ByteArrayInputStream(bytes), fileUploadInfoVO);
    }

    @Override
    public FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception {
        FileUploadConfigMinio minio = fileUploadConfig.getMinio();
        MinioClient minioClient = new MinioClient(minio.getDomain(), minio.getAccesKey(), minio.getSecretKey());
        minioClient.removeObject(minio.getBucket(), fileUploadInfoVO.getFileName());
        return fileUploadInfoVO;
    }
}

存儲工廠類實現

用來根據 FileTag 枚舉類獲得對應的具體實現

@Service
public class FileStorageFactory {

    Map<FileTag, FileStorageService> fileStorageServiceMap;

    @Qualifier("FileStorageServiceLocal")
    @Autowired
    FileStorageService fileStorageServiceLocal;

    @Qualifier("FileStorageServiceSeaweedfs")
    @Autowired
    FileStorageService fileStorageServiceSeaweedfs;

    @Qualifier("FileStorageServiceMinio")
    @Autowired
    FileStorageService fileStorageServiceMinio;

    @PostConstruct
    public void init() {
        fileStorageServiceMap = new HashMap<>();
        fileStorageServiceMap.put(FileTag.TYPE_LOCAL, fileStorageServiceLocal);
        fileStorageServiceMap.put(FileTag.TYPE_SEAWEEDFS, fileStorageServiceSeaweedfs);
        fileStorageServiceMap.put(FileTag.TYPE_MINIO, fileStorageServiceMinio);
    }

    public FileStorageService get(FileTag type) {
        FileStorageService fileStorageService = fileStorageServiceMap.get(type);
        return fileStorageService == null ? fileStorageServiceMap.get(FileTag.TYPE_LOCAL) : fileStorageService;
    }
}

業務中實現

這里根據具體的業務，來選擇具體的實現邏輯
eg：

@Autowired
private FileStorageFactory fileStorageFactory;
public FileUploadInfoVO put(MultipartFile multipartFile) throws Exception {
  FileUploadInfoVO vo = new FileUploadInfoVO();
  // 填充一些信息（省略）
  // 獲得文件上傳處理器
  FileStorageService fileStorageService = fileStorageFactory.get(FileTag.TYPE_LOCAL);
  fileUploadInfoVO = fileStorageService.upload(multipartFile.getInputStream(), vo);
  // ... 業務邏輯
}

這樣，配置文件中可以使用FileTag來控制文件存儲的介質，從而實現一個簡單的對象存儲服務器。