hadoop安裝snappy

Hadoop安裝Snappy
安裝編譯依賴部分：
第一步：安裝Maven
將MAVEN_HOME配置到環境變量里面
需要再settings.xml配置文件中增加以下內容
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
第二步：安裝gcc
yum -y install gcc*
第三步：安裝ncurses-devel，openssl-devel，cmake
yum -y install ncurses-devel openssl-devel cmake
第四步：安裝protobuf
解壓縮：tar -zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0/
./configure
make
make check
make install
第五步：下載snappy源碼
wget ftp://ftp.netbsd.org/pub/pkgsrc/distfiles/snappy-1.1.1.tar.gz
第六步：安裝相關依賴
yum install -y automake autoconf libtool
第七步：編譯snappy源碼
tar -zxvf snappy-1.1.1.tar.gz
cd snappy-1.1.1
./configure
make
make check
make install
編譯完成之后會將snappy相關.so鏈接庫默認安裝到/usr/local/lib目錄下面
編譯hadoop
在編譯的時候，需要指定使用snappy
mvn package -Pdist,native -DskipTests -Dtar -Drequire.snappy
#mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy -Dbundle.snappy -Dsnappy.lib=/usr/local/lib
編譯好之后，將原來的$HADOOP_HOME/lib/native替換為編譯好的新的native庫
然后拷貝編譯后的snappy.so lib庫到$HADOOP_HOME/lib/native/中
編譯hadoop-snappy
mvn package -DskipTests
將編譯好的hadoop-snappy-0.0.1-SNAPSHOT.jar拷貝到$HADOOP_HOME/lib
修改hadoop配置文件
1°、修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh，添加：
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/
2°、修改$HADOOP_HOME/etc/hadoop/core-site.xml：
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec
</value>
</property>
3°、修改$HADOOP_HOME/etc/hadoop/mapred-site.xml中有關壓縮屬性，測試snappy：
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
4°、驗證
hadoop checknative -a