1.准備3台主機
本實例是部署實驗環境，采用的是Citrix的虛擬化環境，分配了3台RHEL6.4的主機。
| 額外需求 | |
|---|---|
| Master | 創建模板后，額外添加20G一塊磁盤/dev/xvdb，額外添加2塊網卡eth1,eth2 |
| Standby | 創建模板后，額外添加20G一塊磁盤/dev/xvdb，額外添加2塊網卡eth1,eth2 |
| Segment01 | 創建模板后，額外添加50G一塊磁盤/dev/xvdb，額外添加2塊網卡eth1,eth2 |
網絡規划
| eth0(外部IP) | eth1 | eth2 | |
|---|---|---|---|
| Master | 192.168.9.123 | 172.16.10.101 | 172.16.11.101 |
| Standby | 192.168.9.124 | 172.16.10.102 | 172.16.11.102 |
| Segment01 | 192.168.9.125(可選) | 172.16.10.1 | 172.16.11.1 |
實驗環境資源有限暫時配置3個節點，后續可能會根據需求添加Segment02，Segment03...
修改主機名
將Master,Standby,Segment01的三台主機名分別設置為mdw, smdw, sdw1
主機名修改方法：
hostname 主機名
vi /etc/sysconfig/network 修改hostname
Options：配置腳本，前期為了方便同步節點間的配置，可選。
export NODE_LIST='MDW SMDW SDW1'
vi /etc/hosts 臨時配置
192.168.9.123 mdw
192.168.9.124 smdw
192.168.9.125 sdw1
配置第一個節點到自身和其他機器的無密碼登錄
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.123
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.124
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.125
cluster_run_all_nodes "hostname ; date"
磁盤規划
gp建議使用xfs文件系統，所有節點需要安裝依賴包
# rpm -ivh xfsprogs-3.1.1-10.el6.x86_64.rpm
所有節點建立/data文件夾，用來掛載xfs的文件系統
mkdir /data
mkfs.xfs /dev/xvdb
[root@smdb Packages]# mkfs.xfs /dev/xvdb
meta-data=/dev/xvdb isize=256agcount=4, agsize=1310720 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=5242880, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
vi /etc/fstab 添加下面一行
/dev/xvdb /data xfs rw,noatime,inode64,allocsize=16m1 1
2.關閉iptables和selinux
cluster_run_all_nodes "hostname; service iptables stop"
cluster_run_all_nodes "hostname; chkconfig iptables off"
cluster_run_all_nodes "hostname; chkconfig ip6tables off"
cluster_run_all_nodes "hostname; chkconfig libvirtd off"
cluster_run_all_nodes "hostname; setenforce 0"
cluster_run_all_nodes "hostname; sestatus"
vi /etc/selinux/config
cluster_copy_all_nodes /etc/selinux/config /etc/selinux/
注：所有節點都要統一設定，我這里先配置了信任，用腳本實現的同步，如果沒有配置，是需要每台依次設定的。
3.設定建議的系統參數
vi /etc/sysctl.conf
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 512000 100 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.core.netdev_max_backlog = 10000
vm.overcommit_memory = 2
kernel.msgmni = 2048
net.ipv4.ip_local_port_range = 1025 65535
vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
同步到各個節點：
cluster_copy_all_nodes /etc/sysctl.conf /etc/sysctl.conf
cluster_copy_all_nodes /etc/security/limits.conf /etc/security/limits.conf
磁盤預讀參數及 deadline算法
在/etc/rc.d/rc.local 添加
blockdev --setra 16385 /dev/xvdb
echo deadline > /sys/block/xvdb/queue/scheduler
cluster_copy_all_nodes /etc/rc.d/rc.local /etc/rc.d/rc.local
注：重啟后 blockdev --getra /dev/xvdb 驗證是否生效
驗證所有節點的字符集
cluster_run_all_nodes "hostname; echo $LANG"
重啟所有節點，驗證修改是否生效：
blockdev --getra /dev/xvdb
more /sys/block/xvdb/queue/scheduler
cluster_run_all_nodes "hostname; service iptables status"
4.在Master上安裝
mkdir -p /data/soft
上傳greenplum-db-4.3.4.2-build-1-RHEL5-x86_64.zip到Master
**解壓**
unzip greenplum-db-4.3.4.2-build-1-RHEL5-x86_64.zip
**安裝**
/bin/bash greenplum-db-4.3.4.2-build-1-RHEL5-x86_64.bin
5.在所有的節點上安裝配置Greenplum
配置/etc/hosts
192.168.9.123 mdw
172.16.10.101 mdw-1
172.16.11.101 mdw-2
192.168.9.124 smdw
172.16.10.102 smdw-1
172.16.11.102 smdw-2
192.168.9.125 sdw1
172.16.10.1 sdw1-1
172.16.11.1 sdw1-2
同步/etc/hosts配置
cluster_copy_all_nodes /etc/hosts /etc/hosts
配置gp需要的互信
vi hostfile_exkeys 創建文件內容示例如下：
mdw
mdw-1
mdw-2
smdw
smdw-1
smdw-2
sdw1
sdw1-1
sdw1-2
Option: 此時如果之前做了部分互信，可以清除之前為安裝方便配置的ssh信任
rm -rf /root/.ssh/
# gpseginstall -f hostfile_exkeys -u gpadmin -p 123456
# su - gpadmin
$ source /usr/local/greenplum-db/greenplum_path.sh
$ cd /usr/local/greenplum-db
$ gpssh -f hostfile_exkeys -e ls -l $GPHOME
互信此時應該是可用的，如果不可用，再次執行
gpssh -f hostfile_exkeys
創建Data Storage Areas，root用戶操作
# mkdir /data/master
# chown gpadmin /data/master/
利用gpssh，在standby master上也創建數據目錄
# source /usr/local/greenplum-db/greenplum_path.sh
# gpssh -h smdw -e 'mkdir /data/master'
# gpssh -h smdw -e 'chown gpadmin /data/master'
在所有的segment節點上創建數據目錄
先創建一個文件 hostfile_gpssh_segonly,包含所有segment節點的主機名
sdw1
創建目錄
# source /usr/local/greenplum-db/greenplum_path.sh
# gpssh -f hostfile_gpssh_segonly -e 'mkdir /data/primary'
# gpssh -f hostfile_gpssh_segonly -e 'mkdir /data/mirror'
# gpssh -f hostfile_gpssh_segonly -e 'chown gpadmin /data/primary'
# gpssh -f hostfile_gpssh_segonly -e 'chown gpadmin /data/mirror'
配置NTP
我這里沒有配置NTP，生產環境建議配置。
驗證OS設置
先建立一個hostfile_gpcheck文件
mdw
smdw
sdw1
驗證
$ source /usr/local/greenplum-db/greenplum_path.sh
$ gpcheck -f hostfile_gpcheck -m mdw -s smdw
20150402:17:56:10:009650 gpcheck:mdw:gpadmin-[INFO]:-dedupe hostnames
20150402:17:56:10:009650 gpcheck:mdw:gpadmin-[INFO]:-Detected platform: Generic Linux Cluster
20150402:17:56:10:009650 gpcheck:mdw:gpadmin-[INFO]:-generate data on servers
20150402:17:56:11:009650 gpcheck:mdw:gpadmin-[INFO]:-copy data files from servers
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[INFO]:-delete remote tmp files
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[INFO]:-Using gpcheck config file: /usr/local/greenplum-db/./etc/gpcheck.cnf
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(None): utility will not check all settings when run as non-root user
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(smdw): on device (xvdd) IO scheduler 'cfq' does not match expected value 'deadline'
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(smdw): on device (xvda) IO scheduler 'cfq' does not match expected value 'deadline'
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(smdw): ntpd not detected on machine
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(sdw1): on device (xvda) IO scheduler 'cfq' does not match expected value 'deadline'
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(sdw1): ntpd not detected on machine
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(mdw): on device (xvda) IO scheduler 'cfq' does not match expected value 'deadline'
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[ERROR]:-GPCHECK_ERROR host(mdw): ntpd not detected on machine
20150402:17:56:12:009650 gpcheck:mdw:gpadmin-[INFO]:-gpcheck completing...
驗證網絡性能
hostfile_gpchecknet_sc1
sdw1-1
hostfile_gpchecknet_sc2
sdw1-2
驗證磁盤I/O和內存
hostfile_gpcheckperf
sdw1
配置本地化設置
字符集的設定
創建初始化文件
$ mkdir -p /home/gpadmin/gpconfigs
$ cd /home/gpadmin/gpconfigs
$ vi hostfile_gpinitsystem
sdw1-1
sdw1-2
拷貝gpinitsystem_config
$ cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config /home/gpadmin/gpconfigs/gpinitsystem_config
$ cd /home/gpadmin/gpconfigs
修改
declare -a DATA_DIRECTORY=(/data/primary /data/primary)
#declare -a MIRROR_DATA_DIRECTORY=(/data/mirror /data/mirror) 以后配置默認就是注釋的
運行初始化工具
$ gpinitsystem -c gpconfigs/gpinitsystem_config -h gpconfigs/hostfile_gpinitsystem -s smdw
初始化過程中報錯：
20150403:10:58:51:032589 gpcreateseg.sh:mdw:gpadmin-[INFO]:-Start Function ED_PG_CONF
20150403:10:58:52:032672 gpcreateseg.sh:mdw:gpadmin-[WARN]:-Failed to insert port=40001 in /data/primary/gpseg1/postgresql.conf on sdw1-2
20150403:10:58:52:032672 gpcreateseg.sh:mdw:gpadmin-[INFO]:-End Function ED_PG_CONF
20150403:10:58:52:032672 gpcreateseg.sh:mdw:gpadmin-[FATAL][1]:-Failed Update port number to 40001
20150403:10:58:52:032589 gpcreateseg.sh:mdw:gpadmin-[WARN]:-Failed to insert port=40000 in /data/primary/gpseg0/postgresql.conf on sdw1-1
20150403:10:58:53:032589 gpcreateseg.sh:mdw:gpadmin-[INFO]:-End Function ED_PG_CONF
20150403:10:58:53:032589 gpcreateseg.sh:mdw:gpadmin-[FATAL][0]:-Failed Update port number to 40000
找到資料：https://support.pivotal.io/hc/communities/public/questions/200372738-HAWQ-Initialization
解決方法：
1.所有節點安裝ed
# rpm -ivh /tmp/ed-1.1-3.3.el6.x86_64.rpm
warning: /tmp/ed-1.1-3.3.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...########################################### [100%]
1:ed ########################################### [100%]
2.清除初始化系統的信息
/bin/bash /home/gpadmin/gpAdminLogs/backout_gpinitsystem_gpadmin_20150403_105721
3.重新初始化系統
gpinitsystem -c gpconfigs/gpinitsystem_config -h gpconfigs/hostfile_gpinitsystem -s smdw
吐槽下：明明依賴ed，官方安裝文檔中卻沒有提及..
安裝成功最后會輸出類似下面的提示：
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:-Greenplum Database instance successfully created
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:-To complete the environment configuration, please
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:-update gpadmin .bashrc file with the following
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:-1. Ensure that the greenplum_path.sh file is sourced
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:-2. Add "export MASTER_DATA_DIRECTORY=/data/master/gpseg-1"
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:- to access the Greenplum scripts for this instance:
20150403:11:13:00:002886 gpinitsystem:mdw:gpadmin-[INFO]:- or, use -d /data/master/gpseg-1 option for the Greenplum scripts
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:- Example gpstate -d /data/master/gpseg-1
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-Script log file = /home/gpadmin/gpAdminLogs/gpinitsystem_20150403.log
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-To remove instance, run gpdeletesystem utility
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-Standby Master smdw has been configured
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-To activate the Standby Master Segment in the event of Master
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-failure review options for gpactivatestandby
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-The Master /data/master/gpseg-1/pg_hba.conf post gpinitsystem
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-has been configured to allow all hosts within this new
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-array to intercommunicate. Any hosts external to this
20150403:11:13:01:002886 gpinitsystem:mdw:gpadmin-[INFO]:-new array must be explicitly added to this file
20150403:11:13:02:002886 gpinitsystem:mdw:gpadmin-[INFO]:-Refer to the Greenplum Admin support guide which is
20150403:11:13:02:002886 gpinitsystem:mdw:gpadmin-[INFO]:-located in the /usr/local/greenplum-db/./docs directory
20150403:11:13:02:002886 gpinitsystem:mdw:gpadmin-[INFO]:-------------------------------------------------------
配置gpadmin環境變量
source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/master/gpseg-1
可選：客戶端會話環境變量
export PGPORT=5432
export PGUSER=gpadmin
export PGDATABASE=gptest
生效並拷貝到standby master
$ source ~/.bashrc
$ scp ~/.bashrc smdw:~/.bashrc
6.創建數據庫gptest
CREATE DATABASE gptest;
$ psql template1
psql (8.2.15)
Type "help" for help.
template1=# help
You are using psql, the command-line interface to PostgreSQL.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
template1=# \h
Available help:
ABORT BEGIN CREATE SEQUENCE DROP OPERATOR CLASS PREPARE
ALTER AGGREGATE CHECKPOINT CREATE SERVER DROP OWNED PREPARE TRANSACTION
ALTER CONVERSIONCLOSE CREATE TABLEDROP RESOURCE QUEUE REASSIGN OWNED
ALTER DATABASE CLUSTER CREATE TABLE AS DROP ROLE REINDEX
ALTER DOMAINCOMMENT CREATE TABLESPACE DROP RULE RELEASE SAVEPOINT
ALTER EXTERNAL TABLECOMMIT CREATE TRIGGER DROP SCHEMA RESET
ALTER FILESPACE COMMIT PREPARED CREATE TYPE DROP SEQUENCE REVOKE
ALTER FOREIGN DATA WRAPPER COPYCREATE USER DROP SERVER ROLLBACK
ALTER FUNCTION CREATE AGGREGATECREATE USER MAPPING DROP TABLE ROLLBACK PREPARED
ALTER GROUP CREATE CAST CREATE VIEW DROP TABLESPACE ROLLBACK TO SAVEPOINT
ALTER INDEX CREATE CONSTRAINT TRIGGER DEALLOCATE DROP TRIGGERSAVEPOINT
ALTER LANGUAGE CREATE CONVERSION DECLARE DROP TYPE SELECT
ALTER OPERATOR CREATE DATABASE DELETE DROP USER SELECT INTO
ALTER OPERATOR CLASSCREATE DOMAIN DROP AGGREGATE DROP USER MAPPING SET
ALTER RESOURCE QUEUECREATE EXTERNAL TABLE DROP CAST DROP VIEW SET CONSTRAINTS
ALTER ROLE CREATE FOREIGN DATA WRAPPER DROP CONVERSION END SET ROLE
ALTER SCHEMACREATE FUNCTION DROP DATABASE EXECUTE SET SESSION AUTHORIZATION
ALTER SEQUENCE CREATE GROUPDROP DOMAIN EXPLAIN SET TRANSACTION
ALTER SERVERCREATE INDEXDROP EXTERNAL TABLE FETCH SHOW
ALTER TABLE CREATE LANGUAGE DROP FILESPACE GRANT START TRANSACTION
ALTER TABLESPACECREATE OPERATOR DROP FOREIGN DATA WRAPPER INSERT TRUNCATE
ALTER TRIGGER CREATE OPERATOR CLASS DROP FUNCTION LISTEN UNLISTEN
ALTER TYPE CREATE RESOURCE QUEUE DROP GROUP LOADUPDATE
ALTER USER CREATE ROLE DROP INDEX LOCKVACUUM
ALTER USER MAPPING CREATE RULE DROP LANGUAGE MOVEVALUES
ANALYZE CREATE SCHEMA DROP OPERATOR NOTIFY
template1=#
template1=# CREATE DATABASE gptest;
CREATE DATABASE
登錄到gptest
$ psql
psql (8.2.15)
Type "help" for help.
gptest=#