https://docs.gitlab.com/ee/administration/high_availability/gitlab.html
https://about.gitlab.com/high-availability/
https://www.oschina.net/translate/gitlab-high-availability?lang=chs
gitlab目錄結構：
find / -name gitlab |grep -v "^/opt/"
/run/gitlab # pid目錄
/etc/gitlab # 配置目錄
/var/log/gitlab # 日志目錄
/var/opt/gitlab # 應用數據保存目錄，要同步的主要是這里的部分數據
/opt/gitlab # bin程序目錄
架構圖：
git1
git2:
（gitlab 允許多個節點，多活）
postgresql集群：
redis集群（哨兵模式）
MFS網絡文件存儲
# MFS集群部署見 http://www.cnblogs.com/linkenpark/p/7416998.html
redis集群（哨兵模式）：http://blog.51cto.com/tianshili/1759289
# PostgreSQL 主從集群部署 http://www.cnblogs.com/linkenpark/p/8339936.html
安裝postgresql擴展pg_trgm
yum -y install postgresql96-contrib-9.6.8 # 默認的 yum -y install postgresql-contrib
創建用戶及庫
su - postgres psql create role gitlab login encrypted password 'pass'; \du ;顯示用戶 create database gitlabhq_production owner=gitlab ENCODING = 'UTF8'; \l ;列出數據庫 ;添加pg_trgm擴展 CREATE EXTENSION pg_trgm;
在MFS服務端根目錄下創建一個項目給gitlab用
mkdir gitlab
cd gitlab; mkdir -p .ssh gitlab-rails/uploads gitlab-rails/shared gitlab-ci/builds git-data
git1、git2 掛載MFS目錄
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
cat /etc/fstab
mfsmount /var/opt/gitlab/.ssh fuse mfsmaster=mfsmaster,mfsport=9421,mfssubfolder=/gitlab/.ssh,mfspassword=passwd,_netdev 0 0 mfsmount /var/opt/gitlab/gitlab-rails/uploads fuse mfsmaster=mfsmaster,mfsport=9421,mfssubfolder=/gitlab/gitlab-rails/uploads,mfspassword=passwd,_netdev 0 0 mfsmount /var/opt/gitlab/gitlab-rails/shared fuse mfsmaster=mfsmaster,mfsport=9421,mfssubfolder=/gitlab/gitlab-rails/shared,mfspassword=passwd,_netdev 0 0 mfsmount /var/opt/gitlab/gitlab-ci/builds fuse mfsmaster=mfsmaster,mfsport=9421,mfssubfolder=/gitlab/gitlab-ci/builds,mfspassword=passwd,_netdev 0 0 mfsmount /var/opt/gitlab/git-data fuse mfsmaster=mfsmaster,mfsport=9421,mfssubfolder=/gitlab/git-data,mfspassword=passwd,_netdev 0 0
git1、git2安裝gitlab
添加gitlab yum源
cat /etc/yum.repos.d/gitlab-ce.repo
[gitlab-ce] name=Gitlab CE Repository baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el$releasever/ gpgcheck=0 enabled=1
yum install -y curl policycoreutils-python openssh-server sudo systemctl enable sshd
systemctl start sshd
firewall-cmd --permanent --add-service=http
systemctl reload firewalld
yum -y install postfix
systemctl enable postfix
systemctl start postfix
yum install -y gitlab-ce
gitlab啟動腳本
cat /etc/systemd/system/gitlab.service
[Unit] Description=gitlab [Service] Type=oneshot RemainAfterExit=yes ExecStart=/bin/gitlab-ctl start ExecStop=/bin/gitlab-ctl stop [Install] WantedBy=multi-user.target
systemctl enable gitlab
git1配置
grep -Ev "^#|^$" /etc/gitlab/gitlab.rb
external_url 'http://git.conf.com'
gitlab_rails['time_zone'] = 'Asia/Shanghai'
# Disable components that will not be on the GitLab application server roles ['application_role'] # Prevent GitLab from starting if MFS data mounts are not available high_availability['mountpoint'] = '/var/opt/gitlab/git-data' # PostgreSQL connection details postgresql['enable'] = false gitlab_rails['db_adapter'] = "postgresql" gitlab_rails['db_encoding'] = "utf8" gitlab_rails['db_database'] = "gitlabhq_production" gitlab_rails['db_username'] = "gitlab" gitlab_rails['db_password'] = "db_password" gitlab_rails['db_host'] = "172.16.3.203" gitlab_rails['db_port'] = 5432 # Redis connection details(單實例、或主從集群) redis['enable'] = false gitlab_rails['redis_host'] = "172.16.3.203" gitlab_rails['redis_port'] = 6379 gitlab_rails['redis_password'] = "redis_password" gitlab_rails['redis_database'] = 0 # Redis connection details(哨兵集群) # redis連接方式只設置一種就可以，CE版也可以連接redis sentinel集群 redis['enable'] = false gitlab_rails['redis_sentinels'] = [ {'host' => 'sentinel-0.conf.com', 'port' => 26379}, {'host' => 'sentinel-1.conf.com', 'port' => 26379}, {'host' => 'sentinel-2.conf.com', 'port' => 26379}, {'host' => 'sentinel-3.conf.com', 'port' => 26379}, ] redis['master_name'] = 'mymaster' redis['master_password'] = 'redis_auth_pass' gitlab_rails['redis_database'] = 0 # nginx enable nginx['enable'] = true
使配置生效
gitlab-ctl reconfigure
如要再次初始化數據，運行 gitlab-rake gitlab:setup（一般前面執行了gitlab-ctl reconfigure已經初始化發數據）
gitlab-ctl start
git2配置
從git1把/etc/gitlab/gitlab-secrets.json 復制到 git2的/etc/gitlab目錄下
touch /etc/gitlab/skip-auto-migrations
grep -v "^#" /etc/gitlab/gitlab.rb
external_url 'http://git.conf.com'
gitlab_rails['time_zone'] = 'Asia/Shanghai'
# Disable components that will not be on the GitLab application server roles ['application_role'] # Prevent GitLab from starting if MFS data mounts are not available high_availability['mountpoint'] = '/var/opt/gitlab/git-data' # disable automatic database migrations gitlab_rails['auto_migrate'] = false # PostgreSQL connection details postgresql['enable'] = false gitlab_rails['db_adapter'] = "postgresql" gitlab_rails['db_encoding'] = "utf8" gitlab_rails['db_database'] = "gitlabhq_production" gitlab_rails['db_username'] = "gitlab" gitlab_rails['db_password'] = "db_password" gitlab_rails['db_host'] = "172.16.3.203" gitlab_rails['db_port'] = 5432 # Redis connection details(單實例、或主從集群) redis['enable'] = false gitlab_rails['redis_host'] = "172.16.3.203" gitlab_rails['redis_port'] = 6379 gitlab_rails['redis_password'] = "redis_password" gitlab_rails['redis_database'] = 0 # Redis connection details(哨兵集群) # redis連接方式只設置一種就可以，CE版也可以連接redis sentinel集群 redis['enable'] = false gitlab_rails['redis_sentinels'] = [ {'host' => 'sentinel-0.conf.com', 'port' => 26379}, {'host' => 'sentinel-1.conf.com', 'port' => 26379}, {'host' => 'sentinel-2.conf.com', 'port' => 26379}, {'host' => 'sentinel-3.conf.com', 'port' => 26379}, ] redis['master_name'] = 'mymaster' redis['master_password'] = 'redis_auth_pass' gitlab_rails['redis_database'] = 0 # nginx enable nginx['enable'] = true
gitlab-ctl reconfigure
gitlab-ctl start
如有更多的 second app 請參照git2配置
# 前端負載均衡
前端再分別對http負載（可用nginx反向代理）
ssh（可用HAproxy代理，或nginx TCP代理）
由其他gitlab服務器數據遷移到此gitlab集群
原gitlab服務器數據備份
可以通過/etc/gitlab/gitlab.rb配置文件來修改默認存放備份文件的目錄
gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
/var/opt/gitlab/backups修改為你想存放備份的目錄即可, 修改完成之后使用gitlab-ctl reconfigure命令重載配置文件即可.
/opt/gitlab/bin/gitlab-rake gitlab:backup:create #執行備份命令
如 /var/opt/gitlab/backups/1524449406_2018_04_23_10.6.4_gitlab_backup.tar
把 備份的數據 1524449406_2018_04_23_10.6.4_gitlab_backup.tar 復制到 gitlab集群的第一個節點的/var/opt/gitlab/backups目錄下
chmod 777 /var/opt/gitlab/backups/1524449406_2018_04_23_10.6.4_gitlab_backup.tar
# 停止相關數據連接服務
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
umount /var/opt/gitlab/gitlab-rails/uploads # 由於恢復時會把該目錄重啟命令成 /var/opt/gitlab/gitlab-rails/uploads.時間戳 ，生產新的/var/opt/gitlab/gitlab-rails/uploads目錄，但該目錄被掛載了會提示設備busy
umount /var/opt/gitlab/gitlab-ci/builds # 由於恢復時會把該目錄重啟命令成/var/opt/gitlab/gitlab-ci/builds.時間戳 ，生產新的/var/opt/gitlab/gitlab-ci/builds目錄，但該目錄被掛載了會提示設備busy
gitlab-rake gitlab:backup:restore BACKUP=1524449406_2018_04_23_10.6.4 #執行恢復備份
mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads_ok
mkdir -p /var/opt/gitlab/gitlab-rails/uploads
mount /var/opt/gitlab/gitlab-rails/uploads
cp -ap /var/opt/gitlab/gitlab-rails/uploads_ok/* /var/opt/gitlab/gitlab-rails/uploads
mv /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds_ok
mkdir -p /var/opt/gitlab/gitlab-ci/builds
mount /var/opt/gitlab/gitlab-ci/builds
cp -rp /var/opt/gitlab/gitlab-ci/builds_ok/* /var/opt/gitlab/gitlab-ci/builds
gitlab-ctl start unicorn
gitlab-ctl start sidekiq
完成恢復
# key認證問題
可能報錯：
報錯情況1：（key沒有從master復制過來）
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:9DdK4jo9LPLg7snd/vueT3wI2dy0hb7CVYRRGOTU8TY.
Please contact your system administrator.
Add correct host key in /c/Users/cd/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /c/Users/cd/.ssh/known_hosts:4
ECDSA host key for git.tuandai888.com has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
報錯情況2：（從master復制過來的key權限沒設置好）
ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
解決方法：
把master的 /etc/ssh/*key* 文件全部復制到從節點來，並注意權限與master的設置一樣，
scp root@[master_IP]:/etc/ssh/*key* /etc/ssh
chown root:ssh_keys ssh_host_ecdsa_key ssh_host_ed25519_key ssh_host_rsa_key
# 注意
只有連接Postgresql master庫, gitlab才能正常使用。因為gitlab web登錄時就要求寫數據庫。連接到postgresql從庫，gitlab web登錄失敗，報500錯。ssh連接倉庫正常。