一、簡介
MogileFS是一個開源的分布式文件存儲系統，由LiveJournal旗下的Danga Interactive公司開發；它主要由三部分組成，第一部分是server端，server端主要包括mogilefsd和mogstored兩個應用程序。mogilefsd實現的是tracker，它通過數據庫來保存元數據信息，包括站點domain、class、host等；mogstored是存儲節點(store node)，它其實是個WebDAV服務，默認監聽在7500端口，接受客戶端的文件存儲請求。在MogileFS安裝完后，要運行mogadm工具將所有的store node注冊到mogilefsd的數據庫里，mogilefsd會對這些節點進行管理和監控。第二部是mogilefs的utils,這部分主要是一些工具，比如mogadm、mogupload、mogfileinfo 等等；第三部分是客戶端API，mogilefs支持眾多編程語言的客戶端API接口，使用對應的客戶端API接口，我們可以編寫mogilefs的客戶端，從而實現來管理mogilefs上的文件；
二、Mogilefs架構圖
提示：tracker的主要作用是代理前端用戶的請求到數據庫查詢相關文件的元信息，同時它還要監控注冊到tracker的storagenode的健康狀態，一旦后端storage掛掉，它會立刻把對應的storage節點上的device從集群信息中刪除（對應數據還在，只是在集群的狀態信息看不到對應設備），上線時再加入到集群信息；客戶端通過向tracker發送請求文件的信息，tracker通過去數據庫中找到對應的文件所在storage節點的ip地址、端口、設備id以及文件fid，然后通過http鏈接的形式從tracker返回給客戶端，客戶端拿到這tracker的返回的文件地址，然后在去對應的storage請求文件，從而在客戶端就訪問到對應的文件內容信息；客戶端和tracker，storage之間通信都是http協議，從本質上講storage就是一個web服務器；所以對於客戶端我們可以使用web服務器的反向代理，也可以是調用mogilefsAPI寫的客戶端程序；
三、Mogilefs安裝部署
環境說明
| 名稱 | 角色 | ip地址 | 端口 |
| node01 | tracker+mysql+storage | 192.168.0.41 | 7001/7500/7501/3306 |
| node02 | tracker+storage | 192.168.0.42 | 7001/7500/7501 |
| node03 | storage | 192.168.0.43 | 7500/7501 |
各server時間同步
提示：如果有必要，可以配置各節點用hosts文件解析；
安裝環境依賴包
[root@node01 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y Loaded plugins: fastestmirror base | 3.6 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package perl-IO-AIO.x86_64 0:4.20-1.el7 will be installed --> Processing Dependency: perl(common::sense) for package: perl-IO-AIO-4.20-1.el7.x86_64 ---> Package perl-IO-stringy.noarch 0:2.110-22.el7 will be installed ---> Package perl-Net-Netmask.noarch 0:1.9015-13.el7 will be installed ---> Package perl-Sys-Syslog.x86_64 0:0.33-3.el7 will be installed --> Running transaction check ---> Package perl-common-sense.noarch 0:3.6-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================= Installing: perl-IO-AIO x86_64 4.20-1.el7 epel 117 k perl-IO-stringy noarch 2.110-22.el7 base 71 k perl-Net-Netmask noarch 1.9015-13.el7 epel 25 k perl-Sys-Syslog x86_64 0.33-3.el7 base 42 k Installing for dependencies: perl-common-sense noarch 3.6-4.el7 epel 28 k Transaction Summary ======================================================================================================================================= Install 4 Packages (+1 Dependent package) Total download size: 283 k Installed size: 636 k Downloading packages: (1/5): perl-Sys-Syslog-0.33-3.el7.x86_64.rpm | 42 kB 00:00:00 (2/5): perl-Net-Netmask-1.9015-13.el7.noarch.rpm | 25 kB 00:00:00 (3/5): perl-IO-AIO-4.20-1.el7.x86_64.rpm | 117 kB 00:00:00 (4/5): perl-common-sense-3.6-4.el7.noarch.rpm | 28 kB 00:00:00 (5/5): perl-IO-stringy-2.110-22.el7.noarch.rpm | 71 kB 00:00:00 --------------------------------------------------------------------------------------------------------------------------------------- Total 642 kB/s | 283 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : perl-common-sense-3.6-4.el7.noarch 1/5 Installing : perl-IO-AIO-4.20-1.el7.x86_64 2/5 Installing : perl-Net-Netmask-1.9015-13.el7.noarch 3/5 Installing : perl-Sys-Syslog-0.33-3.el7.x86_64 4/5 Installing : perl-IO-stringy-2.110-22.el7.noarch 5/5 Verifying : perl-IO-AIO-4.20-1.el7.x86_64 1/5 Verifying : perl-common-sense-3.6-4.el7.noarch 2/5 Verifying : perl-IO-stringy-2.110-22.el7.noarch 3/5 Verifying : perl-Sys-Syslog-0.33-3.el7.x86_64 4/5 Verifying : perl-Net-Netmask-1.9015-13.el7.noarch 5/5 Installed: perl-IO-AIO.x86_64 0:4.20-1.el7 perl-IO-stringy.noarch 0:2.110-22.el7 perl-Net-Netmask.noarch 0:1.9015-13.el7 perl-Sys-Syslog.x86_64 0:0.33-3.el7 Dependency Installed: perl-common-sense.noarch 0:3.6-4.el7 Complete! [root@node01 ~]#
提示：三個節點都需要安裝環境依賴的包；
上傳rpm包到服務器並安裝
[root@node01 ~]# rz rz waiting to receive. zmodem trl+C ȡ 100% 1 KB 1 KB/s 00:00:01 0 Errors.el7.centos.noarch.rpm... 100% 173 KB 173 KB/s 00:00:01 0 Errorsentos.noarch.rpm... 100% 26 KB 26 KB/s 00:00:01 0 Errorsentos.noarch.rpm... 100% 75 KB 75 KB/s 00:00:01 0 Errorsh.rpm... 100% 27 KB 27 KB/s 00:00:01 0 Errors.rpm... 100% 29 KB 29 KB/s 00:00:01 0 Errors.noarch.rpm... 100% 262 KB 262 KB/s 00:00:01 0 Errors [root@node01 ~]# ls MogileFS-Server-2.46-2.el7.centos.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Utils-2.19-1.el7.centos.noarch.rpm [root@node01 ~]# yum install ./*.rpm -y Loaded plugins: fastestmirror Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed ---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed --> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com --> Processing Dependency: perl(DBD::mysql) >= 3 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch ---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed ……省略部分內容…… Installed: MogileFS-Server.noarch 0:2.46-2.el7.centos MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos perl-Perlbal.noarch 0:1.78-1.el6 Dependency Installed: mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7 perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-DBD-MySQL.x86_64 0:4.023-6.el7 perl-DBI.x86_64 0:1.627-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7 perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7 perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7 perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7 perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7 perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7 perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7 perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7 perl-Net-Daemon.noarch 0:0.48-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7 perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7 perl-PlRPC.noarch 0:0.2020-14.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7 perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7 perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7 Complete! [root@node01 ~]#
提示：安裝環境依賴和以上rpm包，首先要配置好epel源；因為有些依賴包，它來自epel源；相關包我已經上傳至我的網盤http://www.qiuhom.com/d/383f34f57cf847beb296/；
安裝mariadb-server
配置yum源
[root@node01 ~]# cat /etc/yum.repos.d/mairadb.repo [mariadb] name=mariadb repo baseurl=https://mirrors.tuna.tsinghua.edu.cn/mariadb//mariadb-10.1.46/yum/centos/7/x86_64/ gpgcheck=0 [root@node01 ~]#
安裝mariadb-server
[root@node01 ~]# yum install MariaDB-server -y Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package MariaDB-server.x86_64 0:10.1.46-1.el7.centos will be installed --> Processing Dependency: MariaDB-client for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: rsync for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: MariaDB-common for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: galera for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Processing Dependency: libjemalloc.so.1()(64bit) for package: MariaDB-server-10.1.46-1.el7.centos.x86_64 --> Running transaction check ……省略部分內容…… Installed: MariaDB-server.x86_64 0:10.1.46-1.el7.centos MariaDB-shared.x86_64 0:10.1.46-1.el7.centos Dependency Installed: MariaDB-client.x86_64 0:10.1.46-1.el7.centos MariaDB-common.x86_64 0:10.1.46-1.el7.centos boost-program-options.x86_64 0:1.53.0-28.el7 galera.x86_64 0:25.3.29-1.rhel7.el7.centos jemalloc.x86_64 0:3.6.0-1.el7 rsync.x86_64 0:3.1.2-10.el7 Replaced: mariadb-libs.x86_64 1:5.5.56-2.el7 Complete! [root@node01 ~]#
提示：mariadb-server包名是大寫的MariaDB-server；
配置mariadb忽略名稱解析
設置mariadb開機自動啟動，並啟動mariadb
添加用於mogilefs訪問數據庫的賬號
初始化數據庫
[root@node01 ~]# mogdbsetup --help
Usage: mogdbsetup [opts]
Options:
Default Description
============ ===========================================
--verbose <off> Be verbose about what's happening.
--dbhost= localhost hostname or IP to database server.
--dbport= dbd default port number to database server.
--dbname= mogilefs database name to create/upgrade.
--dbrootuser= root Database administrator username. Only needed
for initial setup, not subsequent upgrades.
--dbrootpass= <blank> Database administrator password. Only needed
for initial setup, not subsequent upgrades.
--dbuser= mogile Regular database user to create and/or use
for MogileFS database. This is what the
mogilefsd trackers connect as.
--dbpass= <blank> You should change this, especially if your
database servers are accessible to other users
on the network. But they shouldn't be
if you're running MogileFS, because MogileFS
assumes your network is closed.
--type= MySQL Which MogileFS::Store implementation to use.
Available: MySQL, Postgres
--yes Run without questions.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information. Run with --yes to shut up these prompts.
Continue? [N/y]: y
Create/Upgrade database name 'mogilefs'? [Y/n]: y
Grant all privileges to user 'mogilefs', connecting from anywhere, to the mogilefs database 'mogilefs'? [Y/n]: n
Stopped.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information. Run with --yes to shut up these prompts.
Continue? [N/y]: y
[root@node01 ~]#
提示：如果不想讓mogdbsetup在數據庫中創建一個任何地址都可訪問的mogilefs的用戶，在最后的選項中使用n即可，然后在重復執行上述命令即可；如果不想交互式輸入y，也可以直接在命令后面用--yes；
驗證：在數據庫中查看對應的庫是否都已創建，對應庫中是否都有表？
[root@node01 ~]# mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 7 Server version: 10.1.46-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mogilefs | | mysql | | performance_schema | | test | +--------------------+ 5 rows in set (0.01 sec) MariaDB [(none)]> use mogilefs Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [mogilefs]> show tables; +----------------------+ | Tables_in_mogilefs | +----------------------+ | checksum | | class | | device | | domain | | file | | file_on | | file_on_corrupt | | file_to_delete | | file_to_delete2 | | file_to_delete_later | | file_to_queue | | file_to_replicate | | fsck_log | | host | | server_settings | | tempfile | | unreachable_fids | +----------------------+ 17 rows in set (0.00 sec) MariaDB [mogilefs]>
到此數據庫就准備好了
配置mogilefsd連接數據庫
提示：我們只需要修改紅框中的內容即可；db_dsn用於指定連接數據庫的驅動，以及數據庫庫名和主機地址；db_user用於指定連接數據庫所使用的用戶，db_pass用於指定連接數據庫的密碼；listen用於指定mogilefsd監聽的ip地址和端口；后面的配置就是和啟動的進程數量的配置，這個可以根據業務的規模來設置；
啟動mogilefsd
提示：在centos7上啟動mogilefsd時，會報錯，我們可以不用管這個錯誤，實際上mogilefsd是啟動成功的，只要看到對應的端口和進程起來，表示mogilefsd啟動就沒有問題；
配置node01上的mogstored
[root@node01 ~]# cat /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /data/mogilefs [root@node01 ~]# mkdir /data/mogilefs/ -pv mkdir: created directory ‘/data’ mkdir: created directory ‘/data/mogilefs/’ [root@node01 ~]# chown -R mogilefs.mogilefs /data/mogilefs/ [root@node01 ~]# ll -d /data/mogilefs/ drwxr-xr-x 2 mogilefs mogilefs 6 Sep 16 13:37 /data/mogilefs/ [root@node01 ~]#
啟動mogstored
提示：在centos7上啟動mogstored也會同啟動mogilefsd一樣報錯，這個主要是因為systemd的緣故，在centos6不會報錯；到此，node01上的mariadb、mogilefsd和mogstored都已經安裝配置好了；
復制安裝包到node02和node03
在node02上安裝
[root@node02 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO ./*.rpm -y Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed ---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed --> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch ……省略部分內容…… Installed: MogileFS-Server.noarch 0:2.46-2.el7.centos MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos perl-Perlbal.noarch 0:1.78-1.el6 Dependency Installed: mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7 perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-DBD-MySQL.x86_64 0:4.023-6.el7 perl-DBI.x86_64 0:1.627-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7 perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7 perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7 perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7 perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7 perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7 perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7 perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7 perl-Net-Daemon.noarch 0:0.48-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7 perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7 perl-PlRPC.noarch 0:0.2020-14.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7 perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7 perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7 Complete! [root@node02 ~]#
把node01上的mogilefsd的配置文件復制到node02上，然后啟動mogilefsd
把node01上的mogstored配置文件復制到node02上，然后在node02上創建對應的目錄並修改屬主和屬組為mogilefs，然后啟動mogstored
到此node02上的mogilefsd和mogstored就安裝配置啟動好了；
在node03上安裝配置mogstored
[root@node03 ~]# ls MogileFS-Server-2.46-2.el7.centos.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Utils-2.19-1.el7.centos.noarch.rpm [root@node03 ~]# yum install ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm ./perl-*.rpm perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y Loaded plugins: fastestmirror Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed ---> Package MogileFS-Utils.noarch 0:2.19-1.el7.centos will be installed --> Processing Dependency: perl(Compress::Zlib) for package: MogileFS-Utils-2.19-1.el7.centos.noarch ……省略部分內容…… Installed: MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos MogileFS-Utils.noarch 0:2.19-1.el7.centos perl-Danga-Socket.noarch 0:1.61-1.el6.rf perl-MogileFS-Client.noarch 0:1.14-1.el7.centos perl-Perlbal.noarch 0:1.78-1.el6 Dependency Installed: mailcap.noarch 0:2.1.41-2.el7 perl-BSD-Resource.x86_64 0:1.29.07-1.el7 perl-Business-ISBN.noarch 0:2.06-2.el7 perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-Digest.noarch 0:1.17-245.el7 perl-Digest-MD5.x86_64 0:2.52-3.el7 perl-Encode-Locale.noarch 0:1.03-5.el7 perl-File-Listing.noarch 0:6.04-7.el7 perl-HTML-Parser.x86_64 0:3.71-4.el7 perl-HTML-Tagset.noarch 0:3.20-15.el7 perl-HTTP-Cookies.noarch 0:6.01-5.el7 perl-HTTP-Daemon.noarch 0:6.01-8.el7 perl-HTTP-Date.noarch 0:6.02-8.el7 perl-HTTP-Message.noarch 0:6.06-6.el7 perl-HTTP-Negotiate.noarch 0:6.01-5.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-IO-HTML.noarch 0:1.00-2.el7 perl-IO-Socket-IP.noarch 0:0.21-5.el7 perl-IO-Socket-SSL.noarch 0:1.94-7.el7 perl-LWP-MediaTypes.noarch 0:6.02-2.el7 perl-Mozilla-CA.noarch 0:20130114-5.el7 perl-Net-HTTP.noarch 0:6.06-2.el7 perl-Net-LibIDN.x86_64 0:0.12-15.el7 perl-Net-SSLeay.x86_64 0:1.55-6.el7 perl-Sys-Syscall.noarch 0:0.25-10.el7 perl-TimeDate.noarch 1:2.30-2.el7 perl-URI.noarch 0:1.60-9.el7 perl-WWW-RobotRules.noarch 0:6.02-5.el7 perl-libwww-perl.noarch 0:6.05-2.el7 Complete! [root@node03 ~]#
復制node01上的mogstored配置文件到node03上，然后在node03上創建對應的目錄並修改屬主和屬組為mogilefs，然后啟動mogstored
提示：在node03上沒有安裝mogilefs-server這個包，所以在對應系統上沒有mogilefs這個賬號，我們需要手動創建這個賬號；
啟動mogstored
到此三個節點的環境配置，以及服務都已經正常啟動了；
四、使用mogadm管理mogilefs
把三個storage節點注冊到tracker上
提示：如果集群各節點沒有做hosts主機名解析,那么對應的的節點就需要寫清楚ip地址即可，如果端口不是7500，我們還需要用--port去指定端口；
檢查集群
提示：在tracker節點監聽在本機所有地址的7001，執行上述命令不需要加--trackers選項來指明tracker是誰，它默認會去找本機的7001；否則執行上述命令需要加上--trackers來指定tracker的地址和端口；
在非tracker節點執行檢查集群的命令
創建devices
分別在3個mogstored節點的docroot說指定的目錄下創建以dev+數字的目錄，並將屬主和屬組修改成mogilefs
[root@node03 ~]# mkdir /data/mogilefs/dev3 [root@node03 ~]# chown -R mogilefs.mogilefs /data/mogilefs/dev3 [root@node03 ~]# ll -d /data/mogilefs/dev3 drwxr-xr-x 3 mogilefs mogilefs 37 Sep 16 21:02 /data/mogilefs/dev3 [root@node03 ~]#
添加device
查看device
提示：如果在指定節點的docroot所指定的目錄下沒有創建dev+設備編號的目錄，上面是看不到對應設備的空間大小的；
創建domain
創建class
好了，到此我們就可以上傳文件了
上傳文件，把/etc/passwd上傳到files域中，使用html class
提示：mogupload這個工具是mogilefs的一個上傳文件的工具，其中--domain用於指定域，就是把文件放到那個名稱空間中；--class用於指定使用那個class策略來同步文件，默認不指定使用default；--key用於指定客戶端訪問文件的路徑；--file用於指定要上傳的文件路徑；
查看上傳文件的信息
提示：查看上傳文件的信息需要用mogfileinfo這個工具，我們只需要指定trancker，domain以及文件的key就可以看到，該文件的class，devcount,domain,fid,key,length,以及訪問http地址；這里需要說明一下，我們剛才在創建files域中的html這個class 我們指定了mindevcount=3，表示只要使用這個class上傳文件，文件至少有3個副本；而我們現在看到的只有一個，這是因為在centos7上，Sys-syscall這個包的版本過高，導致mogilefs異常；
降級Sys-Syscall，解決centos7上安裝mogilefs 后端mogstored 同步問題
安裝perl編譯環境
[root@node01 ~]# yum -y install make gcc unzip perl-DBD-MySQL perl perl-CPAN perl-YAML perl-Time-HiRes Loaded plugins: fastestmirror base | 3.6 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 2.9 kB 00:00:00 mariadb | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): epel/x86_64/updateinfo | 1.0 MB 00:00:00 (2/2): epel/x86_64/primary_db | 6.9 MB 00:00:00 Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Package gcc-4.8.5-39.el7.x86_64 already installed and latest version Package unzip-6.0-21.el7.x86_64 already installed and latest version Package perl-DBD-MySQL-4.023-6.el7.x86_64 already installed and latest version Package 4:perl-5.16.3-295.el7.x86_64 already installed and latest version Package 4:perl-Time-HiRes-1.9725-3.el7.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package make.x86_64 1:3.82-23.el7 will be updated ---> Package make.x86_64 1:3.82-24.el7 will be an update ---> Package perl-CPAN.noarch 0:1.9800-295.el7 will be installed --> Processing Dependency: perl(local::lib) for package: perl-CPAN-1.9800-295.el7.noarch ……省略部分內容…… Installed: perl-CPAN.noarch 0:1.9800-295.el7 perl-YAML.noarch 0:0.84-5.el7 Dependency Installed: gdbm-devel.x86_64 0:1.10-8.el7 libdb-devel.x86_64 0:5.3.21-25.el7 perl-Digest-SHA.x86_64 1:5.85-4.el7 perl-ExtUtils-Install.noarch 0:1.58-295.el7 perl-ExtUtils-MakeMaker.noarch 0:6.68-3.el7 perl-ExtUtils-Manifest.noarch 0:1.61-244.el7 perl-ExtUtils-ParseXS.noarch 1:3.18-3.el7 perl-Test-Harness.noarch 0:3.28-3.el7 perl-devel.x86_64 4:5.16.3-295.el7 perl-local-lib.noarch 0:1.008010-4.el7 pyparsing.noarch 0:1.5.6-9.el7 systemtap-sdt-devel.x86_64 0:4.0-11.el7 Updated: make.x86_64 1:3.82-24.el7 Dependency Updated: libdb.x86_64 0:5.3.21-25.el7 libdb-utils.x86_64 0:5.3.21-25.el7 Complete! [root@node01 ~]#
提示：在mogstored節點上都要安裝好perl環境依賴包；然后按照下面的步驟重新編譯安裝Sys-Syscall；然后重啟mogstored即可生效；
下載低版本的Sys-Syscall源碼包，下載地址：https://cpan.metacpan.org/authors/id/B/BR/BRADFITZ/Sys-Syscall-0.23.tar.gz；並解壓，進入到源碼目錄
編譯安裝Sys-Syscall
驗證
重啟mogstored，mogilefsd
提示：在某一台tracker節點重新編譯安裝Sys-Syscall，然后重啟mogilefsd和mogstored即可；我這里是在所有節點都編譯安裝了，然后再把所有節點的mogstored和所有的mogilefsd都重啟了一遍，其實用不着；
上傳文件看看對應的副本數量是否還是只有一個？
提示：可以看到對應的devcount和我們之前指定的class上的mindevcount就一樣了；
驗證：訪問指定上述任一一個地址，看看訪問到的文件內容是否是我們上傳到文件內容？
提示：可以看到我們上傳到文件內容在tracker給我們返回的地址上能夠訪問到，並且里面的內容也是一樣的；
驗證：停掉node02上的mogstored，看看對應的tracker是否能夠及時發現對應的storage不在線了呢？
提示：可以看到當node02上的mogstored掛掉以后，在檢查集群，tracker就能及時的發現對應的設備不在線了，這個有點類似nginx對后端web服務器做健康狀態監測的感覺；也就說tracker在監視着注冊在tracker上的storage節點；
到此，mogilefs在centos7上的安裝部署，以及使用mogadm工具管理集群就完成了，對於mogilefs通常我們上傳文件是通過業務程序調用api實現，mogupload只是一個測試工具；對於查看文件，通常用戶也不會使用mogfieinfo返回的鏈接去查看文件，通常會使用一個web反向代理（nginx）去代理用戶的訪問；比如用戶要查看/images/test.jpg，而在tracker返回的對應資源的鏈接中，一定不是/images/test.jpg，這個時候我們就需要用到web反向代理，讓其反向代理能夠幫助我們去tracker取到對應uri的資源鏈接，從而幫助我們去storage節點獲取到對應的文件，然后再返回給用戶；最后還是建議，如果用到mogilefs，最好跑在centos6上，centos7系統環境較高，有些依賴可能成問題；在centos6上，我測試過，完全沒有上述的問題；我們只需要安裝perl-IO-AIO這一個依賴包就可以解決mogstored啟動問題；centos6上需要用到的rpm包可以去我的網盤下載http://www.qiuhom.com/d/47e8f7ac56a445f1bfc6/；