使用flatbuffers

問題

張三是個java程序員,他寫產生數據的程序.李四是個python程序員,他要用python處理張三產生的數據.最直觀常用的方法就是張三用java把產生的數據保存成csv或者xml文件,然后李四用python讀取csv或xml文件.這沒有問題.但現在有一種性能更好的方法,flatbuffers.

作用

可以把flatbuffers理解成一個可執行文件flatc.這個可執行文件可以把表示數據格式的fbs文件轉化成c++,java,python...代碼.張三擅長使用java,他就用flatc把fbs文件轉化成java代碼.用轉化好的java代碼把數據存成flatbuffers的格式.李四用flatc把同樣的fbs文件轉化成自己熟悉的python代碼,李四可以用這份python代碼把flatbuffers格式文件中的數據讀出來.這就實現了不同語言間傳數據.當然這只是一個簡單的使用場景.flatbuffers的作用遠不只是這些.

使用例子

先寫一個表示數據格式的文件school.fbs

namespace school;

table School {
    name:string;
    students:[Student];
}

table Student {
    number:short;
    name:string;
}
root_type School;

具體的語法規則看參考資料,只簡單的解釋下.上面代碼定義了一個根結點為School的數據.里面的student有2個成員變量number,name. School也有2個成員變量,一個是name代碼school的名字.一個是Student的數組.
然后執行下面命令:

flatc.exe -p school.fbs

會生成一堆python代碼.再用生成的python創建一個flatbuffers文件,代碼如下：

# coding:utf-8
from __future__ import print_function
from __future__ import division

import flatbuffers

from school import School
from school import Student

builder = flatbuffers.Builder(1024)

stu1 = builder.CreateString('zhang3')
stu2 = builder.CreateString('li4')

Student.StudentStart(builder)
Student.StudentAddName(builder, stu1)
Student.StudentAddNumber(builder, 3)
zhang3 = Student.StudentEnd(builder)

Student.StudentStart(builder)
Student.StudentAddName(builder, stu2)
Student.StudentAddNumber(builder, 4)
li4 = Student.StudentEnd(builder)

School.SchoolStartStudentsVector(builder, 2)
builder.PrependUOffsetTRelative(li4)
builder.PrependUOffsetTRelative(zhang3)
students = builder.EndVector(2)

School.SchoolStart(builder)
School.SchoolAddStudents(builder, students)
school_result = School.SchoolEnd(builder)

builder.Finish(school_result)

buf = builder.Output()

new_school = School.School.GetRootAsSchool(buf, 0)
print(new_school.Students(0).Name())
print(new_school.Students(0).Number())
print(new_school.Students(1).Name())
print(new_school.Students(1).Number())

with open("result.bin", mode="wb") as f:
    f.write(buf)

這段代碼的作用是新建2個student，一個zhang3,一個li4,然后把student數據保存到School里,最后生成文件result.bin. result.bin就是flatbuffers格式的文件,本質下就是二進制文件.
再執行下面命令:

flatc.exe -j school.fbs

生成一堆java代碼.可以用java代碼把result.bin文件中的數據讀出來.代碼如下:

package com.company;
import school.School;

import java.io.*;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class Main {
    private void process() {
        Path path = Paths.get("D:\\yourpath\\result.bin");
        try {
            byte[] data =  Files.readAllBytes(path);
            ByteBuffer buf = java.nio.ByteBuffer.wrap(data);
            School school = School.getRootAsSchool(buf);
            System.out.println(school.students(0).name());
            System.out.println(school.students(0).number());
            System.out.println(school.students(1).name());
            System.out.println(school.students(1).number());
        } catch (IOException e) {
            System.out.println("IOException");
        }

    }

    public static void main(String[] args) {
        Main m = new Main();
        m.process();
    }
}

另一個名字

google在tensorflow lite中把result.bin這類二進制文件起一個高大上的名字——tflite

參考資料

1. FlatBuffers Documentation
2. flatbuffers源代碼