Hadoop支持LZO

0. 環境准備

maven（下載安裝，配置環境變量，修改sitting.xml加阿里雲鏡像）
gcc-c++
zlib-devel
autoconf
automake
libtool
通過yum安裝即可

yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下載、安裝並編譯LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
tar -zxvf lzo-2.10.tar.gz
cd lzo-2.10
./configure -prefix=/usr/local/hadoop/lzo/
make
make install

2. 編譯hadoop-lzo源碼

2.1 下載hadoop-lzo的源碼

下載地址：https://github.com/twitter/hadoop-lzo/archive/master.zip

2.2 解壓之后，修改pom.xml

<hadoop.current.version>2.7.2</hadoop.current.version>

2.3 聲明兩個臨時環境變量

 export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
 export LIBRARY_PATH=/usr/local/hadoop/lzo/lib 

進入hadoop-lzo-master，執行maven編譯命令

mvn package -Dmaven.test.skip=true

2.5 進入target，將hadoop-lzo-0.4.21-SNAPSHOT.jar放到hadoop的classpath下

如${HADOOP_HOME}/share/hadoop/common

2.6 修改core-site.xml增加配置支持LZO壓縮

<configuration>
        <property>
            <name>io.compression.codecs</name>
            <value>
            org.apache.hadoop.io.compress.GzipCodec,
            org.apache.hadoop.io.compress.DefaultCodec,
            org.apache.hadoop.io.compress.BZip2Codec,
            org.apache.hadoop.io.compress.SnappyCodec,
            com.hadoop.compression.lzo.LzoCodec,
            com.hadoop.compression.lzo.LzopCodec
            </value>
        </property>
        <property>
            <name>io.compression.codec.lzo.class</name>
            <value>com.hadoop.compression.lzo.LzoCodec</value>
        </property>
</configuration>