修復gitlab服務器突然停電導致PostgreSQL損壞的數據庫

最開始是存儲的卷組受損，使用的DRBD，使用了xfs分區格式：

掛載也報錯：

mount /dev/drbd0 /var/opt

mount: wrong fs type, bad option, bad superblock on /dev/drbd0,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)

/var/log/message 報：XFS (drbd0): Corruption warning: Metadata has LSN (152:651864) ahead of current LSN (129:294808). Please unmount and run xfs_repair (>= v4.3) to resolve.

修復卷組：xfx_repair -L /dev/drbd0

 

訪問 gitlab web是報500：

gitlab-ctl tail是看到報錯：

 /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/users/sign_in","format":"html","controller":"SessionsController","action":"new","status":500,"error":"ActiveRecord::StatementInvalid: PG::InternalError: ERROR:  missing chunk number 0 for toast value 127916 in pg_toast_2619\n: SELECT COUNT(count_column) FROM (SELECT  1 AS count_column FROM \"users\" LIMIT 2) subquery_for_count","duration":9.1,"view":0.0,"db":2.41,"time":"2019-07-26T03:16:02.627Z","params":{},"remote_ip":"10.100.17.191","user_id":null,"username":null}

 

這是postgresql查詢users表示報錯

 

訪問GitLab的PostgreSQL數據庫，看看出什么問題了

1.登陸gitlab的安裝服務查看配置文件

cat /var/opt/gitlab/gitlab-rails/etc/database.yml 

production:
  adapter: postgresql
  encoding: unicode
  collation:
  database: gitlabhq_production  //數據庫名
  pool: 10
  username: 'gitlab'  //用戶名
  password:
  host: '/var/opt/gitlab/postgresql'  //主機
  port: 5432
  socket:
  sslmode:
  sslrootcert:
  sslca:

查看/etc/passwd文件里邊gitlab對應的系統用戶

[root@localhost ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
gitlab-www:x:496:493::/var/opt/gitlab/nginx:/bin/false
git:x:495:492::/var/opt/gitlab:/bin/sh
gitlab-redis:x:494:491::/var/opt/gitlab/redis:/bin/false
gitlab-psql:x:493:490::/var/opt/gitlab/postgresql:/bin/sh  //gitlab的postgresql用戶

2.根據上面的配置信息登陸postgresql數據庫

[root@localhost ~]# su - gitlab-psql     //登陸用戶
-sh-4.1$ psql -h /var/opt/gitlab/postgresql -d gitlabhq_production   連接到gitlabhq_production庫

gitlabhq_production=# \dt;   // 查看表

gitlabhq_production=# select * from users; // 報如下錯
ERROR:  missing chunk number 0 for toast value 127916 in pg_toast_2619

 

修復：

參考：https://www.postgresql.org/message-id/CAJfPOeDNcnrPLHm%3DuxO8qLL_g14-QG1O6vyLHvO20oWt0JpPgw%40mail.gmail.com

gitlabhq_production=# select 2619::regclass;
gitlabhq_production=# delete from pg_statistic;
reindex table pg_statistic;
vacuum analyze;

 

就這樣有可以愉快的玩耍了