kafka 配置kerberos校驗以及開啟acl實踐

轉載請注明原創地址:http://www.cnblogs.com/dongxiao-yang/p/7131626.html      

 

   kafka從0.9版本以后引入了集群安全機制，由於最近需要新搭建一套kafka集群，決定采用SASL/GSSAPI (Kerberos) 作為新集群的權限系統基礎，本次新集群版本為0.10.2.0。

   團隊內部已有同學搭建了專門的kerberos服務器，所以省掉了自建kerberos的步驟。

 （1）首先是為broker每台服務器在kerber服務器生成相應的principal和keytab，將下列命令里生成的kafka.keytab文件分發到對應broker機器的統一位置，比如/etc/kafka.keytab

addprinc -randkey kafka/[e-mail]
addprinc -randkey kafka/[e-mail]
addprinc -randkey kafka/[e-mail]
.........


xst -norandkey -k /opt/kafkahost1/kafka.keytab kafka/[e-mail]
xst -norandkey -k /opt/kafkahost2/kafka.keytab kafka/[e-mail]
xst -norandkey -k /opt/kafkahost3/kafka.keytab kafka/[e-mail]
.....

（2）配置kafka server文件

listeners=SASL_PLAINTEXT://:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka //這里的service.name要跟上面建立的principal相對應，kafka/[e-mail]的principal服務名就是kafka
super.users=User:kafka  //acl相關，broker服務本身是采用kafka這個服務身份進行交互，只有配置成superuser才能獲取集群內的metadata信息
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer //acl相關，配置后才能啟用acl

 （3）建立kafka_server_jaas.conf文件，由於集群使用的zookeeper並沒有啟用kerberos，所以沒有client模塊，KafkaClient模塊是為了bin目錄下kafka-console-consumer.sh之類的的腳本使用的

KafkaServer {
            com.sun.security.auth.module.Krb5LoginModule required
            useKeyTab=true
            storeKey=true
            keyTab="/etc/kafka.keytab"
            principal="kafka/[e-mail]";
        };

KafkaClient {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        keyTab="/etc/kafka.keytab"
        principal="kafka/[e-mail]"
        useTicketCache=true;
};

 （4）修改bin目錄下kafka-run-class.sh，在  exec $JAVA 后面增加kerberos啟動參數,然后就可以用正常的腳本啟動服務了：

-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf　　

　

（5）客戶端腳本使用

啟用kerberos后，部分kafka管理腳本需要增加額外的參數才能使用

首先建立配置文件client.properties

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
sasl.mechanism=GSSAPI

所以新命令的使用方式為

bin/kafka-consumer-groups.sh --bootstrap-server kafkahost1:9092 --list --command-config client.properties

bin/kafka-console-producer.sh --broker-list kafkahost1:9092 --topic dxTT --producer.config client.properties

bin/kafka-console-consumer.sh --bootstrap-server kafkahost1:9092 --topic dxTT --consumer.config client.properties

　

 

問題記錄：

kafka服務端配置好kerberos后，controller持續報無法連接到broker的錯誤（包括連接自身實例），大概錯誤如下

[2018-01-25 17:48:41,864] WARN [Controller-60-to-broker-60-send-thread], Controller 60's connection to broker kafka60:9092 (id: 60 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to kafka60:9092 (id: 60 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2018-01-25 17:48:41,970] WARN [Controller-60-to-broker-60-send-thread], Controller 60's connection to broker kafka60:9092 (id: 60 rack: null) was unsuccessful (kafka.controller.RequestSendThread)

 

原因: 打開kafka-authorizer.log的DEBUG日志會看到具體錯誤，這個是由於線上jre的環境缺少kerberos認證的算法庫導致的，更新jre相關類庫即可

[2018-01-25 17:55:31,155] DEBUG Connection with /host disconnected (org.apache.kafka.common.network.Selector)
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]
at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:250)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:71)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:350)
at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
at kafka.network.Processor.poll(SocketServer.scala:494)
at kafka.network.Processor.run(SocketServer.scala:432)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)
at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:235)
... 6 more
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167)
... 7 more
Caused by: KrbException: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled
at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:522)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:273)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829)
... 10 more