Zstd-數據壓縮組件

Zstandard 簡稱Zstd，是一款快速實時的開源數據壓縮程序，由Facebook開發，源碼是用C語言編寫的。相比業內其他壓縮算法（如Gzip、Snappy、Zlib）它的特點是：當需要時，它可以將壓縮速度交換為更高的壓縮比率（壓縮速度與壓縮比率的權衡可以通過小增量來配置），反之亦

Zstd-jni

Zstd擁有豐富的API，幾乎支持所有流行的編程語言，Zstd-jni 是Java中提供的API然。 它具有小數據壓縮的特殊模式，稱為字典壓縮，可以從任何提供的樣本集中構建字典。

<dependency>

    <groupId>com.github.luben</groupId>

    <artifactId>zstd-jni</artifactId>

    <version>VERSION</version>

</dependency>

1.  
   /*
2.  
   序列化
3.  
   */
4.  
   public static <T> byte[] serialize(T result) throws IOException {
5.  
   ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(512);
6.  
   try (OutputStream outputStream = new ZstdOutputStream(byteArrayOutputStream)) {
7.  
   // protostuff serialize
8.  
   ProtostuffSerializer.serialize(result, outputStream);
9.  
   return byteArrayOutputStream.toByteArray();
10.  
    }
11.  
    }
12.  
     
13.  
    /*
14.  
    反序列化
15.  
    */
16.  
    public static <T> T deserialize(byte[] bytes, Class<T> clazz) throws IOException {
17.  
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
18.  
    try (InputStream inputStream = new ZstdInputStream(byteArrayInputStream)) {
19.  
    // protostuff deserialize
20.  
    return ProtostuffSerializer.deserialize(clazz, inputStream);
21.  
    }
22.  
    }

1.  
   ZstdDictTrainer zstdDictTrainer = new ZstdDictTrainer(1024 * 1024, 32 * 1024);
2.  
   // fileInput is a sample file
3.  
   zstdDictTrainer.addSample(fileInput);
4.  
   byte[] dic = zstdDictTrainer.trainSamples(true);
5.  
    
6.  
   /*
7.  
   Zstd's training model
8.  
   */
9.  
   public static <T> byte[] serialize(T result) throws IOException {
10.  
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(512);
11.  
    try (ZstdOutputStream outputStream = new ZstdOutputStream(byteArrayOutputStream)) {
12.  
    if (dic != null) {
13.  
    outputStream.setDict(dic);
14.  
    }
15.  
    // protostuff serialize
16.  
    ProtostuffSerializer.serialize(result, outputStream);
17.  
    return byteArrayOutputStream.toByteArray();
18.  
    }
19.  
    }
20.  
     
21.  
    /*
22.  
    Zstd's training model
23.  
    */
24.  
    public static <T> T deserialize(byte[] bytes, Class<T> clazz) throws IOException {
25.  
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
26.  
    try (ZstdInputStream inputStream = new ZstdInputStream(byteArrayInputStream)) {
27.  
    if (dic != null) {
28.  
    inputStream.setDict(dic);
29.  
    }
30.  
    // protostuff deserialize
31.  
    return ProtostuffSerializer.deserialize(clazz, inputStream);
32.  
    }
33.  
    }

服務器：VM/4CPU i5-6500/CentOS 6

exp1: 4KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	0.162ms	0.059ms	2.31
Snappy	1.1.7.2	0.005ms	0.002ms	1.83
Zstd	1.3.7-3	0.033ms	0.012ms	2.27
Zstd_Dic	1.3.7-3	0.020ms	0.029ms	3.81

exp2: 16KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	0.279ms	0.126ms	4.29
Snappy	1.1.7.2	0.022ms	0.007ms	3.18
Zstd	1.3.7-3	0.049ms	0.002ms	4.39
Zstd_Dic	1.3.7-3	0.062ms	0.017ms	5.70

exp3: 43KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	0.767ms	0.339ms	6.49
Snappy	1.1.7.2	0.111ms	0.044ms	4.36
Zstd	1.3.7-3	0.257ms	0.018ms	6.67
Zstd_Dic	1.3.7-3	0.199ms	0.061ms	8.12

exp4: 134KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	1.786ms	1.026ms	13.34
Snappy	1.1.7.2	0.894ms	0.595ms	6.53
Zstd	1.3.7-3	0.411ms	0.198ms	14.74
Zstd_Dic	1.3.7-3	0.220ms	0.089ms	16.48

exp5:  654KB

	version	compress	decompress	Ratio
Gzip	jdk1.8	4.587ms	1.865ms	33.64
Snappy	1.1.7.2	2.069ms	1.430ms	8.86
Zstd	1.3.7-3	2.864ms	0.116ms	45.57
Zstd_Dic	1.3.7-3	0.426ms	0.218ms	47.38

 

 

引用鏈接：

https://github.com/facebook/zstd

https://github.com/luben/zstd-jni