MogileFS分布式文件系統特點：
1.具有raid的性能
2.不存在單點故障
3.簡單的命名空間： 每個文件對應一個key：用於domain定義名稱空間
4.不共享任何數據
5.傳輸中立，無特殊協議：可以通過NFS或HTTP進行通信
6.自動文件復制：復制的最小單位不是文件，而class
7.應用層： 用戶空間文件系統，無須特殊的核心組件
Nginx+MogileFS的好處：
1、將請求代理至后端MogileFS服務器集群中，能實現負載均衡的效果。
2、能對后端的tracker節點進行健康檢測。
3、將第三方模塊“nginx_mogilefs_module”編譯進Nginx中，能實現直接使用key訪問對應的文件，如下：
使用nginx做代理之前：http://192.168.80.137:7500/dev2/0/000/000/0000000007.fid
使用nginx做代理之后：http://192.168.80.132/image/1.jpg
MogileFS是由三個組件組成的：
1、tracker：MogileFS的核心，是一個調度器，服務進程為mogilefsd，職責：刪除數據、復制數據、監控、查詢等。
2、database：為tracker存儲元素據
3、數據存儲的位置，通常是一個HTTP（webDAV）服務器，用來做數據的創建（put）、刪除（delete）、獲取（get），監聽端口7500, storage節點使用http進行數據傳輸, 依賴於perbal, 進程為mogstored。
理想中模型：
實驗架構：
實驗中用到的rpm包：http://pan.baidu.com/s/1c1S0dJM
MariaDB節點配置：
對各mogilefs節點的主機名進行解析：
[root@www ~]# vim /etc/hosts 192.168.80.136 mog1.daixiang.com 192.168.80.137 mog2.daixiang.com 192.168.80.138 mog3.daixiang.com
二進制包安裝MariaDB:
[root@mariadb mysql]# useradd -r -s /sbin/nologin mysql
[root@mariadb ~]# tar xf mariadb-10.1.14-linux-x86_64.tar.gz -C /usr/local/
[root@mariadb ~]# ln -sv /usr/local/mariadb-10.1.14-linux-x86_64 /usr/local/mysql
[root@mariadb mysql]# chown -R mysql.mysql /usr/local/mysql/
[root@mariadb mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/data
[root@mariadb mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@mariadb mysql]# chmod +x /etc/rc.d/init.d/mysqld
[root@mariadb mysql]# cp support-files/my-large.cnf /etc/my.cnf
[root@mariadb mysql]# vim /etc/my.cnf
datadir = /data
[root@mariadb mysql]# ln -sv /usr/local/mysql/include/ /usr/include/mysql/
[root@mariadb mysql]# vim /etc/ld.so.conf.d/mysql.conf
/usr/local/mysql/lib
[root@mariadb mysql]# vim /etc/profile.d/mysql.sh
export PATH=/usr/local/mysql/bin:$PATH
[root@mariadb mysql]# ldconfig
[root@mariadb mysql]# ldconfig -p | grep mysql
libmysqld.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so.18
libmysqld.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so
libmysqlclient_r.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient_r.so.16
libmysqlclient.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so.18
libmysqlclient.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient.so.16
libmysqlclient.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so
libgalera_smm.so (libc6,x86-64) => /usr/local/mysql/lib/libgalera_smm.so
對用戶進行授權：
MariaDB [(none)]> grant all on *.* to 'root'@'192.168.80.%' identified by 'rootpass'; Query OK, 0 rows affected (0.06 sec) MariaDB [(none)]> grant all on mogilefs.* to 'moguser'@'192.168.80.%' identified by 'mogpass'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> flush privileges; Query OK, 0 rows affected (0.00 sec)
MogileFS配置：
[root@mog1 ~]# yum install *.rpm -y [root@mog1 ~]# yum install perl-IO-AIO -y [root@mog2 ~]# yum install *.rpm -y [root@mog2 ~]# yum install perl-IO-AIO -y [root@mog3 ~]# yum install *.rpm -y [root@mog3 ~]# yum install perl-IO-AIO -y
修改mogilefsd進程的配置文件：
1 [root@mog1 ~]# vim /etc/mogilefs/mogilefsd.conf 2 3 # Enable daemon mode to work in background and use syslog 4 daemonize = 1 #以守護進程的形式運行 5 # Where to store the pid of the daemon (must be the same in the init script) 6 pidfile = /var/run/mogilefsd/mogilefsd.pid 7 # Database connection information 8 db_dsn = DBI:mysql:mogilefs:host=192.168.80.135 #定義數據庫名為mogilefs和數據庫服務器地址 9 db_user = moguser #定義管理此數據庫的用戶名 10 db_pass = mogpass #定義密碼 11 # IP:PORT to listen on for mogilefs client requests 12 listen = 0.0.0.0:7001 #定義監聽的地址和端口 13 # Optional, if you don't define the port above. 14 conf_port = 7001 15 # Number of query workers to start by default. 16 query_jobs = 10 #定義啟動查詢線程個數 17 # Number of delete workers to start by default. 18 delete_jobs = 1 #定義啟動刪除線程個數 19 # Number of replicate workers to start by default. 20 replicate_jobs = 5 #定義啟動復制線程個數 21 # Number of reaper workers to start by default. 22 # (you don't usually need to increase this) 23 reaper_jobs = 1 #響應客戶端請求，在磁盤失敗后將請求重新放到隊列中 24 # Number of fsck workers to start by default. 25 # (these can cause a lot of load when fsck'ing) 26 #fsck_jobs = 1 #對磁盤進行檢測，默認沒有啟動 27 # Minimum amount of space to reserve in megabytes 28 # default: 100 #默認保留100M空間 29 # Consider setting this to be larger than the largest file you 30 # would normally be uploading. 31 #min_free_space = 200 #最小空閑空間為200M 32 # Number of seconds to wait for a storage node to respond. 33 # default: 2 #等待存儲節點相應的時間，默認為2s 34 # Keep this low, so busy storage nodes are quickly ignored. 35 #node_timeout = 2 #等待節點的超時時長 36 # Number of seconds to wait to connect to a storage node. 37 # default: 2 #連接存儲節點的超時時長 38 # Keep this low so overloaded nodes get skipped. 39 #conn_timeout = 2 40 # Allow replication to use the secondary node get port, 41 # if you have apache or similar configured for GET's 42 #repl_use_get_port = 1
設置tracker對應的數據庫，生成mogilefs庫：
[root@mog1 ~]# mogdbsetup --dbhost=192.168.80.135 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=rootpass --dbuser=moguser --dbpass=mogpass --yes #多個tracker共享一個庫，所以其他tracker節點上就不需要在做此操作，直接修改配置文件/etc/mogilefs/mogilefs.conf就可以
注意：此處有一bug，本人也不知道是什么原因造成的：在生成mogilefs庫的時候會出現錯誤，說root用戶沒對mogilefs庫的訪問權限（“Failed to grant privileges: Access denied for user 'root'@'192.168.80.%' to database 'mogilefs'”），但是再一次執行上面的設置數據庫的命令時就會成功。前提是確保mariadb對用戶授權是正確的。如果有哪位大神知道是什么原因造成的，求指點。。
到MariaDB節點上查看mogilefs庫是否成功生成：
[root@Mariadb ~]# mysql
1 MariaDB [(none)]> show databases; 2 +--------------------+ 3 | Database | 4 +--------------------+ 5 | information_schema | 6 | mogilefs | 7 | mysql | 8 | performance_schema | 9 | test | 10 +--------------------+ 11 5 rows in set (0.00 sec) 12 13 MariaDB [(none)]> use mogilefs; 14 Database changed 15 MariaDB [mogilefs]> show tables; 16 +----------------------+ 17 | Tables_in_mogilefs | 18 +----------------------+ 19 | checksum | 20 | class | 21 | device | 22 | domain | 23 | file | 24 | file_on | 25 | file_on_corrupt | 26 | file_to_delete | 27 | file_to_delete2 | 28 | file_to_delete_later | 29 | file_to_queue | 30 | file_to_replicate | 31 | fsck_log | 32 | host | 33 | server_settings | 34 | tempfile | 35 | unreachable_fids | 36 +----------------------+ 37 17 rows in set (0.00 sec)
啟動mogilefsd進程：
[root@mog1 ~]# service mogilefsd start
1 [root@mog1 ~]# ss -tnl | grep '7001' 2 LISTEN 0 128 *:7001 *:*
對mogstored進程進行配置：
[root@mog1 ~]# vim /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /dfs/mogdata
將mogilefsd進程和mogstored進程的配置文件都同步到其他mog節點上去：
[root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.137:/etc/mogilefs/ [root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.138:/etc/mogilefs/ [root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.137:/etc/mogilefs/ [root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.138:/etc/mogilefs/
在存儲節點上創建掛載點：
[root@mog1 ~]# mkdir /dfs/mogdata/dev1
[root@mog1 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
[root@mog2 ~]# mkdir /dfs/mogdata/dev2 -pv
[root@mog2 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
[root@mog3 ~]# mkdir /dfs/mogdata/dev3 -pv
[root@mog3 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/
注意：在生產環境中，需要將存儲磁盤掛載在/dfs/mogdata目錄下，當塊磁盤掛載成功之后，需要在磁盤上創建共享塊設備,即這里博主就不掛載磁盤了，直接創建共享塊設備。
掛載磁盤的配置方法：
[root@mog1 ~]# mkdir /dfs/mogdata/
[root@mog1 ~]# mount -t ext4 /dev/sdb1 /dfs/mogdata/
[root@mog1 ~]# mkdir /dfs/mogdata/dev1
[root@mog1 ~]# chown -R mogilefs,mogilefs /dfs/mogdata/
啟動mogstored進程：
[root@mog1 ~]# service mogstored start [root@mog2 ~]# service mogstored start [root@mog3 ~]# service mogstored start
[root@mog1 ~]# ss -tnlp | grep 'mogstored' LISTEN 0 128 *:7500 *:* users:(("mogstored",2288,4)) LISTEN 0 128 *:7501 *:* users:(("mogstored",2288,9)) [root@mog2 ~]# ss -tnlp | grep 'mogstored' LISTEN 0 128 *:7500 *:* users:(("mogstored",2288,4)) LISTEN 0 128 *:7501 *:* users:(("mogstored",2288,9)) [root@mog3 ~]# ss -tnlp | grep 'mogstored' LISTEN 0 128 *:7500 *:* users:(("mogstored",2288,4)) LISTEN 0 128 *:7501 *:* users:(("mogstored",2288,9))
對各節點進行管理：
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog1 --ip=192.168.80.136 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog1 001 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog2 --ip=192.168.80.137 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog2 002 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog3 --ip=192.168.80.138 --status=alive [root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog3 003 --status=alive [root@mog1 ~]# mogadm domain add linux1 [root@mog1 ~]# mogadm domain add python1 [root@mog1 ~]# mogadm class add linux1 class1 --mindevcount=3 [root@mog1 ~]# mogadm class add linux1 class2 --mindevcount=2 [root@mog1 ~]# mogadm class add python1 dx1 --mindevcount=2
[root@mog1 ~]# mogadm check Checking trackers... 127.0.0.1:7001 ... OK Checking hosts... [ 1] mog1 ... OK [ 2] mog2 ... OK [ 3] mog3 ... OK Checking devices... host device size(G) used(G) free(G) use% ob state I/O% ---- ------------ ---------- ---------- ---------- ------ ---------- ----- [ 1] dev1 16.509 3.901 12.608 23.63% writeable 4.4 [ 2] dev2 16.509 3.901 12.608 23.63% writeable 0.3 [ 3] dev3 16.509 3.897 12.612 23.60% writeable 0.0 ---- ------------ ---------- ---------- ---------- ------ total: 49.527 11.699 37.828 23.62% [root@mog1 ~]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- linux1 class1 3 MultipleHosts() NONE linux1 class2 2 MultipleHosts() NONE linux1 default 2 MultipleHosts() NONE python1 default 2 MultipleHosts() NONE python1 dx1 2 MultipleHosts() NONE
詳細介紹其查看上一篇博客："分布式存儲之MogileFS分布式文件系統簡單應用"
上傳文件進行測試：
[root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=linux1 --key='/1.jpg' --file='/root/centos.jpg' [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key='/1.jpg' [root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=python1 --key='/fstab.html' --file='/etc/fstab' [root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=python1 --key='/fstab.html'
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key='1.jpg' - file: 1.jpg class: default devcount: 2 domain: linux1 fid: 5 key: 1.jpg length: 134783 - http://192.168.80.137:7500/dev2/0/000/000/0000000005.fid - http://192.168.80.136:7500/dev1/0/000/000/0000000005.fid
配置Nginx:
[root@nginx ~]# # yum groupinstall "Development Tools" "Server Platform Deveopment" -y [root@nginx ~]# # yum install openssl-devel pcre-devel -y [root@nginx ~]# useradd -r nginx [root@nginx ~]# tar xf nginx_mogilefs_module-1.0.4.tar.gz [root@nginx ~]# tar xf nginx-1.10.0.tar.gz [root@nginx ~]# cd nginx-1.10.0 [root@nginx nginx-1.10.0]# ./configure \ --prefix=/usr/local/nginx \ --sbin-path=/usr/sbin/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=nginx \ --group=nginx \ --with-http_ssl_module \ --with-http_flv_module \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-client-body-temp-path=/var/tmp/nginx/client/ \ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \ --http-scgi-temp-path=/var/tmp/nginx/scgi \ --with-pcre \ --with-debug \ --add-module=/root/nginx_mogilefs_module-1.0.4 #添加的第三方模塊，實現直接使用key作為uri進行訪問 [root@nginx nginx-1.10.0]# make && make install
提供nginx啟動腳本：
[root@nginx ~]# vim /etc/rc.d/init.d/nginx #!/bin/sh # # nginx - this script starts and stops the nginx daemon # # chkconfig: - 85 15 # description: Nginx is an HTTP(S) server, HTTP(S) reverse \ # proxy and IMAP/POP3 proxy server # processname: nginx # config: /etc/nginx/nginx.conf # config: /etc/sysconfig/nginx # pidfile: /var/run/nginx.pid # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ "$NETWORKING" = "no" ] && exit 0 nginx="/usr/sbin/nginx" prog=$(basename $nginx) NGINX_CONF_FILE="/etc/nginx/nginx.conf" [ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx lockfile=/var/lock/subsys/nginx make_dirs() { # make required directories user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -` options=`$nginx -V 2>&1 | grep 'configure arguments:'` for opt in $options; do if [ `echo $opt | grep '.*-temp-path'` ]; then value=`echo $opt | cut -d "=" -f 2` if [ ! -d "$value" ]; then # echo "creating" $value mkdir -p $value && chown -R $user $value fi fi done } start() { [ -x $nginx ] || exit 5 [ -f $NGINX_CONF_FILE ] || exit 6 make_dirs echo -n $"Starting $prog: " daemon $nginx -c $NGINX_CONF_FILE retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval } stop() { echo -n $"Stopping $prog: " killproc $prog -QUIT retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval } restart() { configtest || return $? stop sleep 1 start } reload() { configtest || return $? echo -n $"Reloading $prog: " killproc $nginx -HUP RETVAL=$? echo } force_reload() { restart } configtest() { $nginx -t -c $NGINX_CONF_FILE } rh_status() { status $prog } rh_status_q() { rh_status >/dev/null 2>&1 } case "$1" in start) rh_status_q && exit 0 $1 ;; stop) rh_status_q || exit 0 $1 ;; restart|configtest) $1 ;; reload) rh_status_q || exit 7 $1 ;; force-reload) force_reload ;; status) rh_status ;; condrestart|try-restart) rh_status_q || exit 0 ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}" exit 2 esac
[root@nginx ~]# chmod +x /etc/rc.d/init.d/nginx
修改nginx配置文件如下：
http { upstream images { server 192.168.80.136:7001; server 192.168.80.137:7001; server 192.168.80.138:7001; } server { listen 80; server_name localhost; location /image { mogilefs_tracker images; mogilefs_domain linux1; mogilefs_methods GET; mogilefs_noverify on; mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } location /files { mogilefs_tracker images; mogilefs_domain python1; mogilefs_methods GET; mogilefs_noverify on; mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } } }
啟動nginx：
[root@nginx ~]# service nginx start
訪問測試：
