目錄
1, 環境
CentOS, MySQL, Redis, Nodejs
2, Redis簡介
Redis是一個開源的K-V內存數據庫，它的key可以是string/set/hash/list/...，因為是基於內存的，所在訪問速度相當快。
3, Gearman簡介
Gearman是一個開源的Map/Reduce分布式計算框架，具有豐富的client sdk，而且它支持MySQL UDF。
Gearman工作圖
Gearman調用流程
Gearman集群
從圖中可以看出貌似Gearman的集群功能比較弱，Job Server之間沒啥關系好像。
4, MySQL - Redis配合使用方案
首先我們以MySQL數據為主，將insert/update/delete交給MySQL，而select交給redis；
當有數據發生變化時，通過MySQL Trigger實時異步調用Gearman的UDF提交一個job給Job Server，當job執行的時候會去更新redis；從而保證redis與MySQL中的數據是同步的。
4, 軟件安裝
安裝gearman
//安裝依賴
$ yum install -y boost-devel gperf libevent-devel libuuid-devel
//下載gearman
$ wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz
$ tar zxvf gearmand-1.1.12.tar.gz
//編譯安裝，指定mysqlclient的鏈接路徑
$ ./configure LDFLAGS=-L/usr/lib64/mysql/
$ make
$ make install
//啟動gearmand服務端
$ /usr/local/sbin/gearmand -l /var/log/gearmand.log -d
//查看是否安裝成功，查看gearman版本
$ /usr/local/sbin/gearmand -V
gearman client
gearman client有很多種：PHP, Perl, Python, Ruby, Nodejs, Go, Java, C#...，我們選擇GearmaNode(Nodejs庫)作為實驗用的gearman client
Nodejs的安裝省略
GearmaNode的安裝及測試
安裝GearmaNode
//安裝GearmaNode
$ npm install gearmanode
啟動Gearman Job Server
//確保gearman的job server已經啟動, job server默認監聽4730端口
$ netstat -nplt | grep 4730
//如果沒有，則啟動job server（-d表示啟動, -l指定log的位置）
$ /usr/local/sbin/gearmand -l /var/log/gearmand.log -d
worker.js (Gearman Job Worker)
var gearmanode = require('gearmanode');
var worker = gearmanode.worker();
//對job client傳遞過來的字符串執行reverse倒序操作
worker.addFunction('reverse', function (job) {
job.sendWorkData(job.payload); // mirror input as partial result
job.workComplete(job.payload.toString().split("").reverse().join(""));
});
//啟動job worker
$ node worker.js
client.js (Gearman Job Client)
var gearmanode = require('gearmanode');
var client = gearmanode.client();
//提交job
var job = client.submitJob('reverse', 'hello world!');
job.on('workData', function(data) {
console.log('WORK_DATA >>> ' + data);
});
//結果回調函數
job.on('complete', function() {
console.log('RESULT >>> ' + job.response);
client.close();
});
//執行client.js
$ node client.js
//查看控制台打印輸出
WORK_DATA >>> hello world!
RESULT >>> !dlrow olleh
可以看到gearman安裝成功，運行正常。
5, MySQL UDF + Trigger同步數據到Gearman
安裝lib_mysqludf_json
lib_mysqludf_json可以把MySQL表的數據以json數據格式輸出
//下載lib_mysqludf_json
$ git clone https://github.com/mysqludf/lib_mysqludf_json.git
$ cd lib_mysqludf_json
//編譯
$ gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c
//拷貝lib_mysqludf_json.so到MySQL的plugin目錄
//可以登陸MySQL，輸入命令"show variables like '%plugin%'"查看plugin位置
$ cp lib_mysqludf_json.so /usr/lib64/mysql/plugin/
//演示lib_mysqludf_json功能
$ mysql -uname -hhost -ppwd
//首先注冊UDF函數
mysql> CREATE FUNCTION json_object RETURNS STRING
SONAME "lib_mysqludf_json.so";
//json_array|json_members|json_values函數注冊方式與json_object一樣.
mysql> use test;
mysql> select * from user_list;
+------+----------+
| NAME | PASSWORD |
+------+----------+
| troy | pwd |
+------+----------+
mysql> select json_object(name,password) as user from user_list;
+----------------------------------+
| user |
+----------------------------------+
| {"name":"troy","password":"pwd"} |
+----------------------------------+
安裝gearman-mysql-udf
//下載
$ wget https://launchpad.net/gearman-mysql-udf/trunk/0.6/+download/gearman-mysql-udf-0.6.tar.gz
$ tar zxvf gearman-mysql-udf-0.6.tar.gz
$ cd gearman-mysql-udf-0.6
//安裝libgearman-devel
$ yum install libgearman-devel
//編譯安裝
$ ./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/
$ make && make install
//登錄MySQL注冊UDF函數
mysql> CREATE FUNCTION gman_do_background RETURNS STRING
SONAME "libgearman_mysql_udf.so";
mysql> CREATE FUNCTION gman_servers_set RETURNS STRING
SONAME "libgearman_mysql_udf.so";
//函數gman_do|gman_do_high|gman_do_low|gman_do_high_background|gman_do_low_background|gman_sum注冊方式類似，請參考gearman-mysql-udf-0.6/README
//指定gearman job server地址
mysql> SELECT gman_servers_set('127.0.0.1:4730');
如果出現異常信息：
ERROR 1126 (HY000): Can't open shared library 'libgearman_mysql_udf.so' (errno: 11 libgearman.so.8: cannot open shared object file: No such file or directory)
表示系統找不到 libgearman.so 文件，一般so都在/usr/local/lib目錄下，修改配置文件/etc/ld.so.conf，將/usr/local/lib目錄加入進去即可:
$ cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
/usr/local/lib
$ /sbin/ldconfig -v | grep gearman*
MySQL Trigger調用Gearman UDF實現同步
DELIMITER $$
CREATE TRIGGER user_list_data_to_redis AFTER UPDATE ON user_list
FOR EACH ROW BEGIN
SET @ret=gman_do_background('syncToRedis', json_object(NEW.name as 'name', NEW.password as 'password'));
END$$
DELIMITER ;
Gearman Nodejs Worker將數據異步復制到redis
redis的安裝步驟省略
啟動redis-server
$ redis-server
安裝redis的Nodejs client:
npm install hiredis redis
redis-worker.js (Gearman Nodejs worker)
var redis = require("redis");
var gearmanode = require('gearmanode');
var worker = gearmanode.worker();
//添加gearman函數syncToRedis
//當MySQL表記錄更改時，此函數會被調用
worker.addFunction('syncToRedis', function (job) {
job.sendWorkData(job.payload);
console.log("-------job.payload: " + job.payload.toString());
//將字符串轉換成json object, 然后調用更新redis
updateRedis(eval('(' + job.payload.toString() + ')'));
job.workComplete("Successed!");
});
//些函數只是簡單的將MySQL表中的一行的記錄按單個字段更新到redis中。可根據實際情況自行擴展
function updateRedis(json)
{
var client = redis.createClient();
client.on("error", function (err) {
console.log("Error " + err);
});
for(var key in json)
{
client.set(key, json[key], redis.print);
}
client.quit();
}
啟動worder:
node redis-worker.js
更新MySQL
mysql> select * from user_list;
+------+----------+
| NAME | PASSWORD |
+------+----------+
| troy | jane |
+------+----------+
mysql> update user_list set name='hello', password='world';
redis-worker控制台輸出：
-------job.payload: {"name":"hello","password":"world"}
查看redis數據
$ redis-cli
127.0.0.1:6379> keys *
1) "password"
2) "name"
127.0.0.1:6379> get name
"hello"
127.0.0.1:6379> get password
"world"
可以看到只要MySQL中user_list表的數據被update了，redis中的數據也會被更新。