想跑https://github.com/ghaughian/mongo-kafka-spark/blob/master/src/pub.py這個程序,發現沒有confluent_kafka庫
1.pip安裝
pip install confluent_kafka
居然報錯:
In file included from confluent_kafka/src/confluent_kafka.c:17:0: confluent_kafka/src/confluent_kafka.h:17:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. error: command 'gcc' failed with exit status 1
上網查,發現需要安裝python-devel和librdkafka-devel
1.查看具體包名
yum list python-devel
python-devel.x86_64
2.安裝
yum install python-devel.x86_64
librdkafka-devel安裝同理。這兩個安裝好后再安裝confluent_kafka就ok了
經驗:如果提示gcc運行時xxx.h沒有,就安裝xxx-devel包!
librdkafka-devel安裝時遇到了點問題。在centos7.3上,我直接用yum install librdkafka-devel.x86_64可以成功,可是在centos6.9(64位)上卻找不到這個包。必須源碼編譯。
參考:http://www.linuxidc.com/Linux/2017-04/142573.htm
1.下載https://github.com/edenhill/librdkafka
2.解壓
unzip librdkafka-master.zip
3.對文件賦權
chmod 777 configure lds-gen.py
4.
./configure
5. 把Makefile.config里面的WITH_LDS=y這一行注釋掉 (64位需要做這一步,否則會報錯/bin/ld:librdkafka.lds:1: syntax error in VERSION script)
6. make
7. make install
在centos6.9上手工安裝了librdkafka並安裝了confluent_kafka后,運行代碼報錯:找不到librdkafka.so.1
解決參考:http://ypf3027.iteye.com/blog/2007743
通過find找到librdkafka.so.1在/usr/local/lib/librdkafka.so.1中
先建立軟鏈
ln -s /usr/local/lib/librdkafka.so.1 /usr/lib
再創建出動態裝入程序(ld.so)所需的連接和緩存文件
sudo ldconfig