这是几年前写的旧文，此前发布Wordpress小站上，现在又重新整理。算是温故知新，后续会继续整理。如有错误望及时指出，在此感谢。
需求描述
在跨平台跨网络的接口访问中，数据的传输往往伴随着序列化和压缩。
在实际项目中，产品与产品之间，服务与服务之间，往往采用不同的技术方案，两者进行数据传输也会因为平台的技术方案不同而出现鸡同鸭讲的情况，数据序列化就是解决这个问题的方案。
序列化的方案很多有JSON，有二进制。
需求目标
- 使用Jdk的Deflater进行数据的对序列化及压缩；
- 使用Protobuf进行数据的序列化及压缩；
- 对比两者的差异；
需求实现
准备了一个应用场景，学校班级里的学生信息进行二进制传输保存。
环境准备
JDK1.8
Maven依赖
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.22</version>
</dependency>
<dependency>
<groupId>io.protostuff</groupId>
<artifactId>protostuff-runtime</artifactId>
<version>1.7.4</version>
</dependency>
<dependency>
<groupId>io.protostuff</groupId>
<artifactId>protostuff-core</artifactId>
<version>1.7.4</version>
</dependency>
Deflater方案
import java.io.*;
import java.util.zip.DataFormatException;
import java.util.zip.Deflater;
import java.util.zip.Inflater;
public class JavaCompressTest5 {
public static void main(String[] args) {
try {
Class clazzObj = productClass();
System.out.println("class对象信息:" + clazzObj.toString());
byte[] objectBytes = object2Byte(clazzObj);
System.out.println("class对象原始大小:" + objectBytes.length);
//压缩
byte[] clazz_data = compresser(objectBytes);
System.out.println("JavaCompress序列化后大小:" + clazz_data.length);
//解压
byte[] decompresser = decompresser(clazz_data);
Class decompressObj = (Class) bytes2Object(decompresser);
System.out.println("decompresser后class对象信息:" + decompressObj.toString());
// class对象信息:Class(id=c_01, name=xiaoban01, students=[Student(id=1, name=zhangsan, age=18), Student(id=2, name=lisi, age=18), Student(id=3, name=wanwu, age=18), Student(id=4, name=zhaoliu, age=18), Student(id=5, name=liuqi, age=18), Student(id=6, name=qianba, age=18), Student(id=6, name=zhoujiu, age=18)])
// class对象原始大小:472
// JavaCompress序列化后大小:320
// decompresser后class对象信息:Class(id=c_01, name=xiaoban01, students=[Student(id=1, name=zhangsan, age=18), Student(id=2, name=lisi, age=18), Student(id=3, name=wanwu, age=18), Student(id=4, name=zhaoliu, age=18), Student(id=5, name=liuqi, age=18), Student(id=6, name=qianba, age=18), Student(id=6, name=zhoujiu, age=18)])
} catch (UnsupportedEncodingException ex) {
// handle
} catch (DataFormatException ex) {
// handle
} catch (IOException e) {
throw new RuntimeException(e);
}
}
/**
* 压缩
*
* @param inputBytes
* @return
*/
public static byte[] compresser(byte[] inputBytes) {
// Compress the bytes
byte[] outputBytes;//存放压缩后的数据
Deflater compresser = new Deflater(Deflater.DEFAULT_COMPRESSION, false);//jdk自带的压缩类Deflater
compresser.setInput(inputBytes);
compresser.finish();
ByteArrayOutputStream baos = new ByteArrayOutputStream(inputBytes.length);
byte[] buf = new byte[100];
try {
while (!compresser.finished()) {
int compressedDataLength = compresser.deflate(buf);//压缩数据写入结果数组的字节数
baos.write(buf, 0, compressedDataLength);
}
outputBytes = baos.toByteArray();
} finally {
baos.hashCode();
compresser.end();
}
return outputBytes;
}
/**
* 解压
*
* @param inputBytes
* @return
*/
public static byte[] decompresser(byte[] inputBytes) throws DataFormatException, IOException {
byte[] outputBytes;//存放解压后的实际数据的数组
Inflater decompresser = new Inflater(false);//解压
decompresser.setInput(inputBytes);//从结果数组中，读取压缩数据
ByteArrayOutputStream baos = new ByteArrayOutputStream(inputBytes.length);
byte[] buf = new byte[100];
try {
while (!decompresser.finished()) {
//实际解压后的结果大小
int inflaterLength = decompresser.inflate(buf);
baos.write(buf, 0, inflaterLength);
}
outputBytes = baos.toByteArray();
} finally {
baos.close();
decompresser.end();
}
return outputBytes;
}
}
@Data
@Builder
class Student implements Serializable {
private String id;
private String name;
private int age;
}
@Data
@Builder
class Class implements Serializable {
private String id;
private String name;
private List<Student> students;
}
Protostuff方案
import lombok.Builder;
import lombok.Data;
import java.io.*;
import java.util.Arrays;
import java.util.List;
public class ProtostuffTest1 {
public static void main(String[] args) {
Class clazzObj = productClass();
System.out.println("class对象信息:" + clazzObj.toString());
byte[] objectBytes = object2Byte(clazzObj);
System.out.println("class对象原始大小:" + objectBytes.length);
//使用ProtostuffUtils序列化
byte[] clazz_data = ProtostuffUtils.serialize(clazzObj);
//System.out.println("pb序列化后：" + Arrays.toString(clazz_data));
System.out.println("Protostuff序列化后大小:" + clazz_data.length);
Class result = ProtostuffUtils.deserialize(clazz_data, Class.class);
System.out.println("Protostuff反序列化后class对象信息:" + result.toString());
// class对象信息:Class(id=c_01, name=xiaoban01, students=[Student(id=1, name=zhangsan, age=18), Student(id=2, name=lisi, age=18), Student(id=3, name=wanwu, age=18), Student(id=4, name=zhaoliu, age=18), Student(id=5, name=liuqi, age=18), Student(id=6, name=qianba, age=18), Student(id=6, name=zhoujiu, age=18)])
// class对象原始大小:472
// Protostuff序列化后大小:122
// Protostuff反序列化后class对象信息:Class(id=c_01, name=xiaoban01, students=[Student(id=1, name=zhangsan, age=18), Student(id=2, name=lisi, age=18), Student(id=3, name=wanwu, age=18), Student(id=4, name=zhaoliu, age=18), Student(id=5, name=liuqi, age=18), Student(id=6, name=qianba, age=18), Student(id=6, name=zhoujiu, age=18)])
}
public static Class productClass() {
Student zhangsan = Student.builder().id("1").name("zhangsan").age(18).build();
Student lisi = Student.builder().id("2").name("lisi").age(18).build();
Student wanwu = Student.builder().id("3").name("wanwu").age(18).build();
Student zhaoliu = Student.builder().id("4").name("zhaoliu").age(18).build();
Student liuqi = Student.builder().id("5").name("liuqi").age(18).build();
Student qianba = Student.builder().id("6").name("qianba").age(18).build();
Student zhoujiu = Student.builder().id("6").name("zhoujiu").age(18).build();
Class clazz = Class.builder().id("c_01").name("xiaoban01").students(Arrays.asList(
zhangsan,
lisi,
wanwu,
zhaoliu,
liuqi,
qianba,
zhoujiu
)).build();
return clazz;
}
public static byte[] object2Byte(java.lang.Object obj) {
byte[] bytes = null;
try {
// object to bytearray
ByteArrayOutputStream bo = new ByteArrayOutputStream();
ObjectOutputStream oo = new ObjectOutputStream(bo);
oo.writeObject(obj);
bytes = bo.toByteArray();
bo.close();
oo.close();
} catch (Exception e) {
System.out.println("translation" + e.getMessage());
e.printStackTrace();
}
return bytes;
}
public static Object bytes2Object(byte[] bytes) {
try {
// object to bytearray
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
ObjectInputStream ois = new ObjectInputStream(bais);
Object obj = ois.readObject();
bais.close();
ois.close();
return obj;
} catch (Exception e) {
System.out.println("translation" + e.getMessage());
e.printStackTrace();
}
return bytes;
}
}
ProtostuffUtils工具类
import io.protostuff.LinkedBuffer;
import io.protostuff.ProtostuffIOUtil;
import io.protostuff.Schema;
import io.protostuff.runtime.RuntimeSchema;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
public class ProtostuffUtils {
/**
* 避免每次序列化都重新申请Buffer空间
*/
private static LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
/**
* 缓存Schema
*/
private static Map<Class<?>, Schema<?>> schemaCache = new ConcurrentHashMap<Class<?>, Schema<?>>();
/**
* 序列化方法，把指定对象序列化成字节数组
*
* @param obj
* @param <T>
* @return
*/
@SuppressWarnings("unchecked")
public static <T> byte[] serialize(T obj) {
Class<T> clazz = (Class<T>) obj.getClass();
Schema<T> schema = getSchema(clazz);
byte[] data;
try {
data = ProtostuffIOUtil.toByteArray(obj, schema, buffer);
} finally {
buffer.clear();
}
return data;
}
/**
* 反序列化方法，将字节数组反序列化成指定Class类型
*
* @param data
* @param clazz
* @param <T>
* @return
*/
public static <T> T deserialize(byte[] data, Class<T> clazz) {
Schema<T> schema = getSchema(clazz);
T obj = schema.newMessage();
ProtostuffIOUtil.mergeFrom(data, obj, schema);
return obj;
}
@SuppressWarnings("unchecked")
private static <T> Schema<T> getSchema(Class<T> clazz) {
Schema<T> schema = (Schema<T>) schemaCache.get(clazz);
if (Objects.isNull(schema)) {
//这个schema通过RuntimeSchema进行懒创建并缓存
//所以可以一直调用RuntimeSchema.getSchema(),这个方法是线程安全的
schema = RuntimeSchema.getSchema(clazz);
if (Objects.nonNull(schema)) {
schemaCache.put(clazz, schema);
}
}
return schema;
}
}
结论
首先这里没有对序列化本身所花费的时长进行统计，有兴趣的同学可以自行统计。
单从结果来看，Protostuff序列化后的大小是Jdk的Deflater的2.6倍，如果数据样本再多些，可能还会更高。
自此，我们使用Protostuff帮我们完成了数据的序列化和压缩两个问题。
希望对有同样需求的同学有帮助。