大數據基礎之Doris（1）編譯安裝和啟動

一 編譯

doris編譯有兩種方式，一種是docker編譯，一種是直接裸機編譯，推薦使用docker編譯，可以避免大量的環境依賴問題

docker編譯

1 安裝docker

yum install docker
systemctl start docker
systemctl enable docker
docker pull apachedoris/doris-dev:build-env-1.2

2 下載源碼

wget https://mirrors.bfsu.edu.cn/apache/incubator/doris/0.13.0-incubating/apache-doris-0.13.0-incubating-src.tar.gz
tar xvf apache-doris-0.13.0-incubating-src.tar.gz

3 啟動容器

docker run -it -v /root/.m2:/root/.m2 -v /path/to/apache-doris-0.13.0-incubating-src:/root/apache-doris-0.13.0-incubating-src apachedoris/doris-dev:build-env-1.2

需要做兩個目錄映射，一個是maven的repository目錄，一個是doris源碼目錄，避免容器掛了之后之前下載或編譯的內容丟失

4 編譯doris

cd /root/apache-doris-0.13.0-incubating-src
sh -x build.sh

編譯之后輸出至output目錄，有3個子目錄：be、fe、udf，只需要拷貝output目錄到其他服務器即可

編譯時報錯

[ERROR] Plugin net.sourceforge.czt.dev:cup-maven-plugin:1.6-cdh or one of its dependencies could not be resolved: Failed to read artifact descriptor for net.sourceforge.czt.dev:cup-maven-plugin:jar:1.6-cdh: Could not transfer artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh from/to spring-plugins (https://repo.spring.io/plugins-release/): Authentication failed for https://repo.spring.io/plugins-release/net/sourceforge/czt/dev/cup-maven-plugin/1.6-cdh/cup-maven-plugin-1.6-cdh.pom 401 Unauthorized -> [Help 1

修改如下：

                 <!-- for java-cup -->
                 <repository>
                 <!--
                     <id>cloudera-thirdparty</id>
                     <url>https://repository.cloudera.com/content/repositories/third-party/</url>
                     -->
                     <id>cloudera-public</id>
                     <url>https://repository.cloudera.com/artifactory/public/</url>
                 </repository>


                 <!-- for cup-maven-plugin -->
                 <pluginRepository>
                 <!--
                     <id>spring-plugins</id>
                     <url>https://repo.spring.io/plugins-release/</url>
                     -->
                     <id>cloudera-public</id>
                     <url>https://repository.cloudera.com/artifactory/public/</url>
                 </pluginRepository>

5 編譯broker

cd fs_brokers/apache_hdfs_broker
sh -x build.sh

6 編譯spark connector

cd extension/spark-doris-connector
sh -x build.sh

裸機編譯

1 准備

jdk8+
maven

sudo yum groupinstall 'Development Tools' && sudo yum install maven cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk

其中：centos7上gcc默認4.8.5，cmake默認2.8.12

2 升級GCC

yum install gcc-c++
wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-7.3.0/gcc-7.3.0.tar.gz
tar zxvf tar zxvf gcc-7.3.0.tar.gz
cd gcc-7.3.0
yum install lbzip2
./contrib/download_prerequisites
mkdir build
cd build/
../configure --enable-checking=release --enable-languages=c,c++ --disable-multilib
make
make install

3 升級CMAKE

wget https://cmake.org/files/v3.6/cmake-3.6.2.tar.gz
tar xvf cmake-3.6.2.tar.gz && cd cmake-3.6.2/
./bootstrap
gmake
gmake install
mv /usr/bin/cmake /usr/bin/cmake.bak
ln -s /usr/local/bin/cmake /usr/bin/

4 編譯

sh -x build.sh

報錯處理

報錯1

Downloading libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip from https://doris-incubating-repo.bj.bcebos.com/thirdparty/libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip to /usr/local/app/doris/thirdparty/src
--2021-01-11 09:32:59-- https://doris-incubating-repo.bj.bcebos.com/thirdparty/libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip
Resolving doris-incubating-repo.bj.bcebos.com (doris-incubating-repo.bj.bcebos.com)... 2409:8c00:6c21:10ad:0:ff:b00e:67d, 220.181.33.44, 220.181.33.43
Connecting to doris-incubating-repo.bj.bcebos.com (doris-incubating-repo.bj.bcebos.com)|2409:8c00:6c21:10ad:0:ff:b00e:67d|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-01-11 09:32:59 ERROR 404: Not Found.

因為url已經修改，參考git上最新的文件如下：
https://github.com/apache/incubator-doris/blob/master/thirdparty/vars.sh

解決方法：將url修改為

LIBEVENT_DOWNLOAD="https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip"

參考：https://github.com/apache/incubator-doris/issues/5519

報錯2

下載第三方DataTables報錯403，直接訪問
https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.22
提示

Error: Only libraries of the current release version can be used by the. package builder. The library DataTables's current release version is 1.10.24. Please reload the download builder page to have it use the latest libraries.

解決方法：修改thirdparty/vars.sh，將版本改為最新的1.10.24，同時修改md5sum

# datatables, bootstrap 3 and jQuery 3
#DATATABLES_DOWNLOAD="https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.22"
#DATATABLES_NAME="DataTables.zip"
#DATATABLES_SOURCE="DataTables-1.10.22"
#DATATABLES_MD5SUM="62558846fc6a6db1428e7816a2a351f7"
DATATABLES_DOWNLOAD="https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.24"
DATATABLES_NAME="DataTables.zip"
DATATABLES_SOURCE="DataTables-1.10.24"
DATATABLES_MD5SUM="22404292d02cf3c5f4cd9f5a02d4b42c"

報錯3

checking how to run the C preprocessor... /usr/lib64/ccache/../bin/cpp
configure: error: in `/data/app/apache-doris-0.13.0-incubating-src/thirdparty/src/unixODBC-2.3.7':
configure: error: C preprocessor "/usr/lib64/ccache/../bin/cpp" fails sanity check

解決方法：

ln -s /usr/local/bin/cpp /usr/lib64/bin/cpp

報錯4

./comp_err: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

解決方法：

yum -y install libatomic

報錯5

cp: cannot stat ‘./zstd_ep-install/lib/libzstd.a’: No such file or directory

解決方法：

cd thirdparty/src/zstd-1.3.7/tests
make zstd-staticLib
mkdir -p /path/to/apache-doris-0.12.0-incubating-src/thirdparty/src/arrow-apache-arrow-0.15.1/cpp/release/zstd_ep-install/lib64
cp ../lib/libzstd.a /path/to/apache-doris-0.12.0-incubating-src/thirdparty/src/arrow-apache-arrow-0.15.1/cpp/release/zstd_ep-install/lib64

二 啟動

1 啟動FE

cd output/fe
mkdir doris-meta
bin/start_fe.sh --daemon

日志在log目錄下

注意：

• 默認8030端口可能與yarn的resourcemanager沖突
• 啟動之后檢查fe綁定端口的ip是否正確，如果綁定ip錯誤（比如安裝docker之后取到docker的ip），會導致be無法連接fe，需要配置fe.conf中的priority_networks，配置為正確的網段

priority_networks=192.168.1.0/24

2 啟動BE

cd output/be
mkdir /path/to/storage
vim conf/be.conf
bin/start_be.sh --daemon

日志在log目錄下

注意：

• 默認8040端口可能與yarn的nodemanager沖突
• 如果be啟動失敗，一般可能有兩個原因：一個是端口被占用，一個是limit，根據日志排查，比如報錯
  Doris Be http service did not start correctly. exiting.
  則是因為端口占用導致web啟動失敗

修改limit

臨時修改

limit -n 65535

永久修改

vim /etc/security/limit.conf
*               hard    nofile             65535
*               soft    nofile             65535

3 啟動Broker

cd fs_brokers/apache_hdfs_broker/output/apache_hdfs_broker
bin/start_broker.sh --daemon

三 使用

1 FE web訪問

http://fe_server:8030

注意：

• 端口為http_port
• 默認賬號root，密碼空

2 FE 命令訪問

mysql -P9030 -uroot
show proc '/frontends';

注意：

• 端口為query_port

3 添加或刪除FE

alter system add follower '$host:$port';
show proc '/frontends';

4 添加或刪除BE

alter system add backend '$host:$port';
alter system dropp backend '$host:$port';
show proc '/backends';

注意：

• port默認為9050，即heartbeat_service_port

5 添加或刪除Broker

alter system add broker $broker_name '$host:$port';
show proc '/brokers';

注意：

• port默認為8000，即broker ipc_port

四 數據導入

hive數據導入

create database test;
CREATE TABLE test.test_user_doris
(
  `id` varchar(128) , 
  `name` varchar(128) , 
  `country` varchar(128) , 
  `province` varchar(128) , 
  `city` varchar(128) ,  
  `order_count` int SUM
)
AGGREGATE KEY(id, name, country, province, city)
DISTRIBUTED BY HASH(id) BUCKETS 10
PROPERTIES("replication_num" = "3");

LOAD LABEL load_test_user_doris
(
    DATA INFILE("hdfs://nameservice1/user/hive/warehouse/test.db/test_user/*")
    INTO TABLE `test_user_doris`
    FORMAT as "parquet"
)
WITH BROKER $broker_name 
(
    "dfs.nameservices" = "nameservice1",
    "dfs.ha.namenodes.nameservice1" = "namenode1,namenode2",
    "dfs.namenode.rpc-address.nameservice1.namenode1" = "nn1:8020",
    "dfs.namenode.rpc-address.nameservice1.namenode2" = "nn2:8020"
)
PROPERTIES
(
    "timeout"="36000",
    "max_filter_ratio"="0.1"
);

show load;

五 參數配置

show variables;

同mysql

參考：

• 編譯 http://doris.apache.org/master/en/installing/compilation.html
• 啟動 http://doris.apache.org/master/en/installing/install-deploy.html